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Section 1 
XPraRODUCTIOU 


1.1 STUDY OBJECTIVE 

The problem of producing reliable software has "been ^fith us for a number 
of years yet until recently only portions of the problem have been 
attacked. For example, automated tools for analysing program structures 
have been developed and loudly acclaimed by some while others promote 
tools for automated test case generation. The use of a theoretical approach 
to the problem has also been investigated but at no time has a comprehensive 
view of the total problem been taken. The purpose of this study is to 
perform for NASA an investigation into the areas having an impact on 
producing reliable software including automated verification tools , software 
modeling, testing techniques, structured programming and management 
techniques. This final report contains the results of this investigation, 
analysis of each technique, and the definition of a methodology for pro- 
ducing reliable software. 

Task I - Automated Verification Tools (Appendix A) 

. Investigation was made into the existence, availability and 
applicability of automated verification tools such as PET, 

ATDG, FORTUNE, etc, 

. A comparative analysis was made and relative merits evaluated. 

, Recommendations regarding development of new tools and modifi- 
cations of the existing tools for NASA applications are 
included. 

Task II - Software Modeling (Appendix B) 

. Evaluation of the existing approaches to software modeling 
was made. The practical application of statistical techniques 
to software quality measurement was assessed, 

. Correlation between programming structures and figure-of-raerit 
indexes is considered among the major recommendations for 
future research. 
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Cask III - Frogn-ani Testing (Appendix C) 

. TeEting teclmiq.ues currently availatle were eveilxiated.. 

. Problems of test case design were considered. 

. Techniques for test design and optimization in relation to 
. cost and software reliability are recommended for future 
research. 

Task IV - Structured Programming (Appendix D) 

. State-of-the-art structured programming applications were 
studied with specific consideration of higher level 
languages suitability to structured programming. 

Task V - Program Management Techniques (Appendix P) 

. Chief Programmer Techniques and Computer Program Management 
Techniques were studied and comparative analyses are made. 

Task VI - Proving Pro gr ams Correct (Appendix E) 

. A review of literature on the techniques of Proving Program 
Correctness was made. A glossary of terms with relevance 
to Software Reliability was developed. Techniques which 
may be applicable to Test Case Selection will be identified. 

Final Report - Overview 

This final report contains: an evaluation and assessment of the practic- 

ability of each available technique, a description of a methodology for 
producing reliable software., suggestions for automating portions 
of the methodology, and a comprenensive bibliography. 
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1.2 STUDY RECOMMENDATIONS 

In surveying the various tools and techniques cijrrently heing used or 
developed for producing more reliable software, one quickly realizes 
the need for an overall methodology. A great deal of work has been put 
into various tools promising increased software reliability. At the 
same time a few well known individuals have been espousing the 
belief that by applying certain management and organization concepts 
programmers will produce error free programs . Despite these sometimes 
inflated claims, both the tool builders and the organization advocates 
have made positive contributions towards achieving more reliable 
software systems. However, it is now quite clear that an integrated method- 
ology including sound organizational concepts and procedures with complemen- 
tary support from automated tools offers an improvement. 

With this philosophy in mind, this final report attempts to present an 
overall methodology for producing reliable software. Tools are incorporated 
into this methodology at numerous points. A series of appendices contain 
a significant amount of data on currently available tools and techniques. 

The test tool survey contained in Appendix A is a "first”. It is the 
first time that an attempt has been made at comparing a number of 
currently available tools and providing the reader with a means of 
examining the capabilities of all of the tools. A number of obvious 
omissions can be observed due to the reluctance of some tool builders 
to include their tools in such a survey. One very definite recommendation 
of this study is to expand this survey to include additional tools. 

Another area suggesting more research is the development of more meaningful 
measures for comparing the cost and performance associated with the use 
various tools. 

With respect to the types of tools currently available, it is strongly 
recommended that static analyzer tools and dynamic execution analyzer 
tools be incorporated into a facility's general support software. 

These tools should be made available for general use and in connection 
with the suggested methodology contained in this report. 

Two distinct areas of research mentioned in this report appear very promising 
with regard to tools. Appendix C describes the use of an embedded assertion 
language for bridging the gap between requirements specification and the 
functional testing of those specifications. This approach promises to 
malte a major impact on the actual programming process. Appendix C also 
contains an interesting look into a promising means for rigidly examining 
various propel ties of a program and thereafter heing able to informally 
prove the corresponding correctness of the selected properties. Additional 
research in these a."eas can lead to more powerful tools in the future. 

Software modeling is another research area demanding additional effort. 

In particular, the refinement of error data collection process, the assoc- 
iation of the error data with the original source of the anomaly, and 
the correlation of the degree of testing applied to a module of software 
and the number of errors associated with that module constitute ongoing 
research topics. 
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1.3 METHODOLOGY OVERVIEW 

In most systems Leing deTeloped today, the quality of the software is often 
the limiting factor of the total system (Quality. Rigorous quality assTirance 
disciplines have been imposed on hardware for many years resulting in 
highly reliable hardware. These same disciplines imposed on the software 
have not achieved reliability of the software because, with the state-of- 
the-art tools, software cannot he tested as rigorously as hardware. 

Most often the end product is still less than satisfactory. 

The problem is often the result of several factors: 

. quality assuring activities are imposed late in the development 
cycle 

. quality assuring activities are treated as unrelated activities 

. quality assuring activities are not always the first concern 
of the software builder. 

In the normal course of software development, the initial requirements 
analysis and development specifications are both incomplete and conflicting. 
If the inadequacies are not discovered and corrected, they are incorporated 
into the design. The poor design is then implemented, in code. The problems 
are not detected until the testing phase which reveals the need for changes 
to the detailed design which in turn may cause changes in the requirements. 
This recognition of the problem late in the development cycle results in 
software redesign that is often diffictilt to incorporate into existing work, 
causing costly overruns. 

Ideally, this iterative interaction of error detection and correction should 
be confined to successive phases. When this is time, the goal of ensuring 
the con^jleteness of the requirements is pursued during the design phase. 

The requirements then should he completely and accurately defined and 
under formal controls by the time testing begins. 

The achievement of quality mftware can be promoted by the application of 
a methodology that imposes quality producing activities on the 
development cycle. 

This methodology consists of three interdependent sets of techniques: 

1) software production techniques 

2) software verification/validation techniques 

3) software management techniques 

Software production techniques include such items as: 

. top down development 
. structiired programming 
. use of a program design language (PDL) 

. use of tools such as compiler writers, meta-assemblers and 
language preprocessors where applicable. 





Software verification/validat ion techniques include such direct activities 
as : 

. requirements analysis and feedhack 

. an assertion methodology for placing specification checks within the. code 
. code verification using walk throughs, standards checkers, 
execution analyzers , f lowpath analyzers and debugging aids . 

. program validation hy module, acceptance and system integration 
testing, 

. program certification 

Software management techniques are the means "by which the development is 
ordered and controlled, and hy which visibility is provided into the 
status of progress and quality of the software development. These 
techniques include: 

. configuration management 

. program library control 

. use of stringent documentation standards 

However, the presence of these activities does not in Itself assure that 
quality will be achieved. The decisions on when to begin an activity 
and on the degree of the discipline with which it will be performed are 
crucial. Decisions that shape the project have an increasing influence 
on the development cycle as it progresses. ^Then quality considerations 
are delayed in the development cycle as shown in Figure 1-1, their 
influence is limited. By the time coding and testing have started and 
project members are concerned with product quality, their efforts to 
achieve that quality are often restricted by past decisions. 

Figure 1-2 shows the effect of including quality considerations in the 
criteria for the software decisions in all phases of development, so that 
each phase contributes to the quality goal. Since the development of a 
software system is basically a human activity, it is imperative that 
information about the deveopment is constantly and clearly fed back to the 
developer and the user right from the beginning. 

The immediate imposition of standards to assure quality documentation 
provides confidence that the intent of the documents is accurate3y 
communicated to those who must rely on those documents. 

Early review and analysis of specifications allows early decisions on 
redesign so that changes are preventive rather than cxirative. The test 
and evaluation function begins with specification analysis and const^tly 
impacts all phases. Early considerations of requirements testability 
reduces likelihood of costly impacts in the later phases of development. 

The early determination of requirements for production aad verification 
tools allows time for them to be procured (built, purchased or leased) 
and checked out before they are needed. Reliability measurement definitions 
at an early stage dictate the data to be collected and applied during 
development . 
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CONSIDERATIONS 













Configuration management anci program library control processes instituted 
early reduce confusion in th.e detailed design and coding phases and 
allow development to progress in an orderly fashion. 

The traditional approach to software quality assurance has been to have 
the quality functions such as testing, configuration management and 
program library control be performed by the software builder. There 
are several inherent disadvantages to this approach. The most obvious 
disadvantages are the development priority structure of the builder, 
which is different than the priorities of an independent evaluator, and 
the use of personnel to whom the quality assurance functions are of 
secondaiy interest. 

Independent evaluation and monitoring provides an unbiased and effectual 
approach to software quality assurance. This assurance can he 
provided by imposing a quality producing methodology for controlling 
and validating the software at every stage of its development. 

The functions required to achieve reliable software must he viewed as 
totally interrelated functions. The system cannot he used with confidence 
unti. it is well tested. It cannot he well tested without a comprehensive 
analysis of the requirments. The verification of requirements using 
test tools cannot be performed adequately without well checked out tools 
which must he identified in the requirements stage. The testing also is 
virtually meaningless without assurance of the integrity of the program 
library, which depends to a large extent on the effectiveness of the 
configuration management of the software. And finally, good documentation 
is required to provide feedback, visibility and assurance that these 
functions are effectively implemented. 

The formulation of a methodology to produce reliable software for MSA 
Goddard considers the interrelationship of quality producing functions, 
the decisions that cause the imposition of these functions at the proper 
time, and the enforcement of the standards and controls that maintain 
the benefits of each function. 
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Section 2 
TERMINOLOGY 


Througliout the literatiire hearing on the subject of reliable soft-ware, there 
is considerable disagreement on the meaning of several key terms. To 
provide a consistent base for understanding this document, the folio-wing 
definitions have been selected for these terms. Although the list may 
appear to be elementary, it* is intended to eliminate some of the ambiguity 
of meaning. 

Software - the computer programs with their associated 

data bases, job control language and documen- 
tation. 

the set of properties of the software that 
characterize how well it works, how easy it 
is to use, and how easy it is to maintain. 

(definition by Schick and Nolverton ) ”the 
probability that the applications program, 
together with its supeiTrisory program, data 
bases, and computing environment -will perform 
its intended functions at the time when those 
functions are needed by the customer*' . 

Verification - The process by which the set of specifications 

and/or axioms describing the natiire of a problem 
and its environment are checked for completeness, 
consistency, and systematically compared with 
the resulting software representation. 

Verification addresses functional correctness 
and usually involves a great deal of manual 
effort. Automation of some of the comparative 
and analytical functions are currently being 
researched. 

Validation - The process of inspecting software behavior 

in the operational en-vironment (i.e. , hardware, 
software, data sources, man-machine interface) 
and determining that the software will in fact 
perform its proper function. Vfhile verification 
attempts to build convincing arguments for the 
''correct** representation of a problem, validation 
addresses the questions: 


Software quality 


Software reliability - 
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Debugging 

Development testing - 

Unit testing 
Module testing 

Integration testing - 
Operational testing - 

Acceptance testing 

Baseline testing 

Program Proving 


1) Is the software really solving the right problem? 

2) From an operational point of view is the 
software useable? 

the process of finding and correcting errors that 
are syntactic or structural in nature {not 
specifically associated with the verification and 
validation of basic program f\inGtions) and 
that prevent successful execution of the software. 

confirmation by the programmer that a completed 
software module performs as intended using a 
trivial test case. It is an informal test 
carried out in the software configuration 
currently being developed. 

generally synonomous with development testing. 

confirmation that a completed module performs 
as intended when subjected to a comprehensive 
set of test cases in the software configuration 
currently being developed, 

confirmation that a total hardware /softwax’e system 
performs as intended when the entire system 
is executed in a test environment. 

the process of validating the system by exercising 
it for a given period of time in a user 
environment, using test procedures designed to 
exercise it comprehensively. 

the testing performed with a set of test cases 
designed to verify that the completed system 
performs in accordance with specified acceptance 
criteria, 

confirmation that a completed system continues 
to meet the critical requirements during 
maintenance of the software. The test 
cases used are designed to exercise all software 
critical to the system performance. 

The set of formal, mathematical and informal 
quasi-mathematical techniques, often semi-automated, 
for checking the consistency of program specifications, 
axioms, and program code. Program proving is often 
equated with verification in academic circles. 

For purposes of this study, however, verification 
will he defined to include program proving 
techniques together with other less mathematically 
rigorous techniques for checking consistency and 
completeness . 


2-2 


Certification 
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the process of placing an authpritatiYe staa^ of app 
roval on a software system. Certification 
implies that a software system has heen 
subjected. to an ade(iuate level of testing and 
recommenas its operational usage. Certification 
should include adequate verification and 
validation of the, software although in current 
practice this is often not done. 


Reference 

1, G. J, Schick, R. W. Wolverton Assessment of Software Reliahility,. . 
Proceedings of German Operations Research Society, September 1912 , 





Section 3 

SOFTWARE MANAGEMENT TECHNIQUES 


3,1 GENERAL 

The effectiTe management of software development is of primary importance in 
producing reliable software. There are several techniques that -^ll provide 
the controls that help guarantee orderly and efficient development. 

Most of these techniques are used in some form at NASA Goddard, This .section 
discusses enhancements of the techniques currently used that may offer 
greater visibility and more reliability. 

The Erogranmier’s Reference Manual provides standards and conventions for pro.- 
gramming and documentation to ensure consistency of the product. It provides 
information to the programmer about the use of the system hardware and soft- 
ware that is installation-pc-culiar , . and includes the detailed use of procedures 
that maintain control of configured lihraries. 

It also contains the procedures for designing effective development tests to 
be performed by the programmer. 

Configuration management is discussqd in relation to NASA’s current procedures. 
A technique for maintaining automeitic configuration status is offered using 
tools that will allow traceability of changes to the statement level* The 
capabilities of this of tool include those already existing in the up- 

dating program used hy NASA Goddard. The enhanconents that provide the addi- 
tional capahilities ar'e described in detail in this section. 

The establishment, updating, release, and maintenance of controlled libraries 
are affected by the techniques that affect configuration management. While 
the program library control and configuration management • are discussed separ- 
ately, they are interdependent. 

The discussion of the software development organization stresses the use of 
two techniques: l)’ independent analysis, review, testing. and evaluation, 

and 2) -^the. team approach to software development. 

The last paragraphs discuss some techniques that help in organizing and' 
measuring the dfevelopraent of the project. The selection of tools and 
techniques and the appropriate, time to apply these tools and techniques 
are part of establishing the goals for project development. These tools 
^and techniques are used both in the software production and software veri- 
fication/validation areas. 
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3.2 SOFTWARE DOCUMENTATION 


3.2.1 Programmer's Referenee Manual 

The programmer's reference manual contains the information a programmer 
needs to Icno-w about the environment in which he must work that is facility 
oriented. The purpose of this maiaual is to supplement information supplied 
in vendor manuals. The programmer's referenee manual serves several functions. 

1. It provides information about the resources available within the 
computer facility. These resources include the hardware configur- 
ation, the utility libraries, and the computer room and scheduling 
operations . 

2. It contains the standards and conventions to he followed to insure 
standardization and completeness of the finished software projects. 
This includes programming standards and conventions, and documenta- 
tion standards. 

3. It prescribes the procedures to be followed in using the working 
and controlled libraries, and the procedures for installing new 
and modified code in a controlled envirorment . 

U, It describes the recommended tools, and techniques to be used in 
checking out the code. This includes debugging aids, static and 
dynamic analysis tools, and development test techniques. 

NASA Goddar*d has published a software standards guide to be used by in-house 
and contractor personnel using the IBM S60 facility. This d<^cument contains 
some of the information that should be present in a programmer's reference 
manual. This methodology recommends other information for inclusion. 

1. Software Development Notebooks should be used and maintained by 
the programmer and the librarian. This technique is discussed 
in detail in Section 3.2.2. 
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2. Top-Down Development is a design technique to be used for organizing 
the development of a system. It is discussed in detail in Appendix D. 

3. Tools and technic ues are available that will aid FORTRAN code to be 
structured. These tools'and techniques are discussed in detail in 
Appendix D. 

4. The use of software production, testing, and documentation tools 
should be Included. Candidate tools are discussed in Appendix A. 


■ ■mil' 
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5. The prograraiaer ' s reference manual should contain a section out- 
lining the procedures to he followed to insure that a meaningful 
development test has heen designed and performed. Development 
testing is discussed in detail in Section 4.3.1. 

6, The development and maintenance of a -well- controlled software 
system requires that the programmer understand how to effectively 
use the working and controlled libraries contaii^ing the system 
being developed or in operation. Program library control is 
discussed in Section 3.3. 

T. This methodology recommends the team approach to program develop- 
ment. The responsibilities of each member of the team and his 
interfacing requirements should be included in the programmer's 
reference manual. The team approach to program development is 
discussed in Appendix D. 

3.2.2 Software Development Notebooks 

The use of Software Development Notebooks forces atttntion to every aspect 
of the development of software routines- The notebotv'c provides a guide 
to and a record of specific programming activities ant is used to assist 
in program documentation. 

The notebook concept has been implemented at M)AC and TRW for the Site 
Defense Program (DSP) software development with a great deal of success. 

TEW calls the notebook "Unit Development Folder’* (UDF) . 

. . 1 

R. D. Williams describes the advantages of using their UDF as more than 
being a collection point for all pertinent development and test information. 
It ensures that documentation is updated as part of the development activity. 
It serves as a tool to help. foresee impending difficulty in time to avoid 
it. It helps in obtaining meaningful estimates through direct involvement 
of project personnel in scheduling their own work, in accomplishing con- 
tinuous mor'toring and accurate reporting, in avoiding a proliferation of 
phantom problems described by Brooks^a and in making effective use of 
time in updating plans or initiating corrective action at a time when it 
can do the most good. 

3. 2. 2.1 Software Development Notebook Standards 

A Software Development Notebook should comply with the following standards: 

. Each module developed requires a notebook. 

. Initially, each notebook will be assigned to one programmer. 

. In late stages of program development, more than one programmer 
may have simultaneous responsibility for a notebook (module). 

. A programmer may have simultaneous responsibility for more than 
one notebook. 
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. Changes to items 1-5 (see Fignre 3~l) of a notebook at any time 
requires authorised, approval. 

. Changes to all other items in a software notebook after the 
routine has been placed in a controlled library requires 
authorized approval. 

3. 2.2.2 Sofware Develcpment Notebook Contents 


Figure 3-1 shows the cover sheet of a sample software development notebook. 



















Table 3-1 

DEVELOPMEKT NOTEBOOK CONTENTS 


1. Requirements Specification 


2. Design Description 


3. F’cctional Flowchart 


U. Interface Definition 


5. Assumptions and Constraints 


6. Module Code 


T, Development Test Case 
Descriptions 


This is the written material that de- 
scribes the requirements on the 
routine; it tells what the routine 
shall do and how well it must do it. 

This is an English language descrip- 
tion of how the routine shall perform. 

It is a description of the design 
that is being proposed to satisfy 
the requirements specification of 1 
above . 

A flow diagram of the design described 
in 2 above. 

A list of all externally provided 
inputs and all generated output destined 
for use by other routines. 

A description of how the routine is 
invoked, how the called routines are 
used, the timing constraints, estimated 
core requirements and any unique con- 
ditions or assumptions associated 
with the routine. 

The initial routine code (a listing) 
which will be updated throughout the 
development period and which will keep 
pace with the code throughout its 
development . 

A description of all development test 
cases which are to be used to checkout 
the routine and the results that can 
be expected from these test cases. 
Testing should be based upon both the 
functional capabilities list and the 
requirements specification of 1 
above. Expected test case results 
will be included in the folder in 
advance of running tt tests. 
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DEVELOPMENT NOTEBOOK CONTENTS (Continued) 


8. Code and Test Case Revie>r 

9. Test Case Results 

10. Detailed Plow Diagram. 

11. Updating Design Description 

12. Program Library Control 

13. Discrepancy Report File 
lU. Sign-off Completed Routine 


An engineering review (ty someone other 
than the developer) to determine 
that the routine code will perform 
as defined in 1 above and that the 
proposed test cases do satisfactorily 
demonstrate this capability. This 
review is held in advance of actual 
testing. 

A compilation of all test case 
results to demonstrate that the 
routine is debugged and that routine 
development testing is complete. 

A listing of the debugged routine 
is to be included. 

One which thoroughly details the 
delivered routine. 

An updated/revised item 2 (corres- 
ponding to item 10 above) . 

The point at which the completed code 
is entered into the controlled 
library. This is at the completion 
of the development testings after 
the code and test case review. 

Copies of every discrepancy report 
that req.uired modification of the 
routine, or its documentation. They 
are added as they are resolved, 

A formal acknowledgement that the 
routine is accepted for installation. 
For both new and modified systems , 
this is after formal qualification 
test, at the time the system is 
released for operation by Release 
Control. 


3. 2. 2. 3 EstaTslishing, Maintaining and Using the Uotehook 

The procedure for preparing and using the notebook is as follows: 

1. The routines are assigned with a budget allocation and final 
delivery date. The appropriate reviews are then allocated. 

2. Each routine assigned to a programmer /analyst has its own 
notebook with a cover sheet. At the time of assignment 
the progre^muer negotiates his final date. Any schedule 
discrepp^icies are resolved and the resolved final date 

is entered opposite the item on the notebook cover sheet, 

"Updated Design Description" (see Figure 3-1). 

3. The programmer is then required to plan his activity and 
schedule the various other items on the cover sheet. His 
supervisors now have incremental visibility into all the 
development steps of the software development activity. 

k. The notebooks are always kept in one of two places, a file 
in the programming office area or the programmer's desk. 

If on the programmer’s desk, then they will be signed out 
from the manager's file. 

5. The notebooks will be available for review by authorised 
personnel at any time that they are not in use by the 
programmer . 

6. The programmer has the responsibility to update the cover of 
the notebook to reflect the current status of its contents. 

The programming manager has the responsibility to review the 
contents and approve the cover sheet. 

3.3 SOFTWARE CONTROL 

Software control is achieved by the implementation of three interrelated 

disciplines. 

l. Configuration management which is the .day-to-day monitoring and 
control of the computer program configuration items (CPCi's) of 
which a system is composed. These items include the computer 
program and the associated documentation. 

2. Release Control which is the process of closing out and turning over 
the software and related documentation. Software that is released 
consists of the computer program (source and object code) and 
supporting computer listings. The related documentation consists 
of the functional and detailed specifications and the User's 
Manuals. Each release establishes a new baseline for the product 
against which future modifications are made. 
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Program library com:rol -vrhicli is tlie process of establishing and 
maintaining a program, library in -which every statement is 
known, documented, and traceable to the justification for its 
existence. It forms the basis for the configuration management 
of the code, the testability of the code, and the assurance that 
the integrity of released code is achieved. 

The following paragraphs discuss these disciplines in reverse order. 

3.3.1 Program Library Control 

GSFC currently maintains three types of program libraries requiring control. 

1. A permanent , distributable library containing all programs 
developed at GSFC. 

2. On-line libraries containing soirrce code, load modules, data bases, 
etc., of operational on-line systans, 

3. A source and document library setup for developing large pieces of 
software. 

The first two lihraries contain only completed code already released. The 
third library is established and used during development. 

This methodology discusses the recommended system for program library control 
that builds upon the system currently used at GSFC. It is supported by a 
tool that performs the functions that are performed by the utility in use 
at GSFC to automatically update and compile the programs , but in addition 
maintains the configuration status of the soft-ware and provides traceability 
of each statement back to the justification for its existence. One such 
tool has been developed for the HQ Space and Missile Organization (SAMSO) 
at Los Angeles and could be made available for use at GSFC. 

3. 3.1.1 Working Libraries and Controlled Libraries 

During the development of a system, the necessity to continually work -with 
existing baselined code requires the co-existence of working and controlled 
lihraries. With the implementation of top-do-wn development, this is true 
even with systems for which the greatest part is still in the form of 
duffiity stubs. 

The establishment of a controlled library consists of defining each I’outine, 
macro, data base segment, etc., as a configured item, assigning unique names 
and configuration identifiers, initializing the release and modification 
number of the tested code, and creating a permanent, protected library. 

Once the code is placed on the controlled library, it is established as 
baselined. Master and critical copies of the baselined code are created. 


The use of the recofflniended update/compile prograia allows programmers to 
easily* create working libraries for the development and debugging of 
new program segments or modifications to existing program segments. 

Selected parts of the baselined source may be copied, updated and stored 
•in a working library 3 either from TSO or from card decks* The listing 
provided by the update program should contain the configuration status 
information and identiiy the source of justification for the code (e.g, , 
problem reports or design specs) . 

The programmer is forced to he aware of the version of the code with which 
he is working. He must input the current configuration revision symbol 
and is r.jturned an error message if it is incorrect. This helps insure 
that he is not updating a different set of code than he thinks he is. 

A parallel copy of the baselined library is maintained by a central group 
whose function is to maintain decks, memos, documentation, etc., and serve 
as the focal point for information about the program status. This library 
should be controlled to the extent that all changes and additions must be 
reviewed and approved before entry, documented both internally and externally, 
the configuration status maintained, and indepth testing performed against 
a known configuration. 

The programmer's private working library is uncontrolled. The update pro- 
gram allows him to easily create a new file for new code or to copy into 
a working data set the code to he modified. He may then develop his code, 
debug the affected routines, link the new routines to the parallel system, 
perform development tests, and execute against a benchmark test without 
disturbing the basic system. 

Control begins when the new approved code is placed into the parallel 
library. 

At the point of making a foimial modification and release, the current pro- 
cedure is to freeze the update in the parallel library, test the frozen 
system, and when accepted, the new library is renamed and placed into 
operation. 

If the changes are significant in number and/or complexity, it is recommended 
that the new system be completely rebuilt at the time of the update freeze. 
This means that the old system is copied to a new library, and all changes 
(including new and deleted routines) are made at one time to this library. 
This insures that every line of code in the new library is known, accounted 
for, and justified. It removes the possibility of a change of code being 
made that is undocumented or unjustified. The update program places all 
of the changes for a mod into a mod packet and writes them into a file on 
the end of the tape containing the updated source. This way, every change 
making up a modification is permanently stored in a readily retrievable 
fom. In addition to the advantage of maintaining this record for historical 
purposes, it allows easy reconstruction of a change that must be backed 
out if an error is found after the formal mod is made. 
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3. 3. 1.2 Release Control . 

’fhe release of a controlled system, implies that it has "been rigorously 
tested, completely documented, its configuration confirmed, all approvals 
have heen obtained, and a master copy of the system (in both source and 
object form) placed in bond by a quality assurance organization . 

The release function forma*^.izes the installation of a new or modified 
system. It assures that the catalogued procedures are updated to reflect 
the new release, thereby minimising the .'.nadvertant use of a prior library 
when future changes are to be made or tests are to be performed. 

The release control function is also concerned with protection of the 
operational system. Release control personnel should be the only 
personnel authorized to obtain a master copy of the system from Q.A. 
bond and restore the syste!« onlipe. 

A Data Release Authorisation is issued at uhe time of release and identi- 
fies the following: 

1. Sequence number of release items 

2. Mnemonic name of volume or document number 

3. Ctirrent revision number 

4. Security classification 

5. Volume number, e.g, , tape reel number 

6. Program identification number 
T. Next higher level of program 
8. Responsible signatures 

3.3.2 Configuration Management 

Software configuration management is the day— to-day monitoring and control 
of the computer program configuration items (CPCI's) of which the system 
is composed. These items include the computer programs and the associated 
document at i on . 

The normal configuration management discipline is applied to the software. 
This discipline consists of four functions: 

. configuration identificati* ■ 

. configuration change control 
. configuration status accounting 
. configuration audit 

3 . 3 . 2 . 1 Conf igurat ion Ident if i cat ion 

Configuration identification is a system of computer program identification 
numbers and document identification numbers that will identify all config- 
urable items. 

The configuration of a computer program should be documented in the specifi- 
cations. The required configuration is identified in the design specifica- 
tion, and the achieved configuration is identified in the post development 
documentation. 
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Ideirtifiaation of tlie configurable software items for use with (or without) 
the update/compilation program consists of a comprehensiTe identification 
scheme. The following example shows one proposed method for accomplishing 
this goal: 

System - A one to eight character name followed by a 

one character release number and one character 
modification number. 

Programs - A filename of one to eight characters defining 

a file of one or more card decks. 

A deckname of one to eight characters defining 
a deck containing a subroutine, macro, data base, 
etc. 

A revision symbol of two characters heginning with 
AA. 

- Listings show the same configuration identifica- 
tion as the programs. 

Documents - A one to eight character program document identifier 

followed hy a two character revision symbol. 

Documents to he assigned configuration identification numbers include: 

, Part I CPCI (Functional) Specification 
. Part XI CPCI (Detailed Design) Specification 
. Interface Specification 
, Software Development Notebooks 

3.3. 2.2 Configuration Change Control 

Configuration Change Control applies to all changes to configured software 
and documentation after the configured items are initially released. 

All proposed changes to approved baselines are assessed, reviewed, and 
evaluated by the Change Control Board (CCB) . The CCB is composed of 
representatives of the Design Group. 

Actions hy the CCB include verifying compliance with contractual require- 
ments and assuring the identification, evaluation and consideration of the 
technical reasons for the change(s). The CCB guarantees that only those 
changes for which a requirement exists, or which offer a significant 
benefit to the program, are initiated. Members of the CCB determine the 
impact that proposed changes will have in the areas of cost, production, 
reliability, maintainability, producihility, logistical support and 
specifications. 


Proposed changes are evaluated as they are proposed. The changes to he 
incorporated are recorded and collected together until the xipdate 
freeze before the formal mod. At that time all incorporated changes 
are submitted as a unit to the CCB for final approval. 

Proposed changes are initiated on a software problem report. The soft- 
ware modification report records the fix cr improvement that was made. 

An engineering change proposal is used to formally submit the collected 
changes to the CCB for final approval. A specification change notice is 
used to record changes to specifications. 

The CCB meets on a periodic basis or when a major change or improvement 
must be evaluated. These meetings are supplemented with separate 
individual meetings for review and action on individual problems that 
must be addressed. 

Minutes of CCB meetings contain the transactions and assigned action 
items. They are not authorizations for changes s but are for historical 
and administrative purposes. 

The problem reports being reviewed and any instruction for their disposi- 
tion are attached to the minutes ^ signed by the CCB chairman, and distri- 
buted to all CCB members plus any others affected by the decisions. 

All proposed changes must be addressed by the CCB. The decision to implement 
or not implement the change is made and recorded. The responsibility 
for investigating any unresolved changes is assigned by the CCB to a 
person who will obtain the information necessary for a decision to made. 

Changes procedures apply to documentation as well as code. 

3. 3. 2. 3 Configuration Status Accounting 

Configuration status accounting is the recording and reporting of the 
status of the system’s configuration. The ptirpose is to know exactly 
what the current configuration is, and how it was achieved. 

Configuration status accounting includes reporting and recording; 

. the initial configuration identification 
. the proposed changes to the configuration 
, all approved changes to the configuration 

As the initial configuration identification is updated, records are main- 
ta.ined that provide traceability of software problems or improvements 
from their inception through the corrective action to their incorporation 
into the existing system. 

Status accounting applies to all controlled program dociimentation, the 
software development notebooks, and the code. Logs are maintained on 
the receipt, identification and disposition of all change forms. These 
inc3 ade software problem reports , enginee ring change requests and 
sp.;Cification change notices. 
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Status reports are published periodically. The information is retrieved 
automatically by report generators from the files maintained on a mass 
storage volume . 

Other files contain the: 

, history of all revisions and changes to each specification 
, history and content of all problem reports 

The status repox-ts contain: 

. history and status of specif icaticn changes 
. summary of the status of all problem reports 
acted upon by the COB, including disposition 
and schedixle of change. 

. status of any individual status report 
, listing of the current computer program configuration 

The above status reports can be obtained in a number of formats using 
various sorting criteria. This provides optimum visibility in any 
desired area. 

3.3.2.i} Configuration Audit 
Configuration audit consists of 

1. a series of reviews of the requirements, the design, and the 
final baseline qualification, and 

2. an audit of the functional and physical configuration of the 
system. 

The purpose of the specifics bion reviews is to confirm the presence of the 
information, clearly and accurately stated, necessary to develop the software 
that meets the requirements. Any problems de bee ted are documented and 
presented for resolution. 

The reviews serve to systematically evaluate the developing system and 
the end product in respect to its conformance with the requirements. 

Baselines are estabilished for the requirements, the design and the 
implemented code and documentation. The documentation is reviewed to 
assure that it accurately describes the product. 

The audits assure that the configuration of the system is compliant with 
the requirements both functionally and physically. 

3.4 SOFTWARE DEVELOPMENT ORGANIZATION 

This methodology recommends two approaches to software development 
organisation, 

1. independent review, analysis and evaluation of the requirements 
and design, and independent test and evaluation of the 
completed product. 
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2. a team approach to software development. 

The concept of the performance of 'verification/validation and control 
functions (which include the verification/validation of the requirements 
and the design) "by personnel who did not specify, design, or hulld the 
system and who are specifically skilled in the areas of analysis and 
evaluation is hecoming more widely accepted. There are two 3Biajor advantages. 
One is the elimination of the logical bias inherent in having the designer/ 
implementer perform these functions. The other is that the functions 
are performed hy personnel to whom this discipline is of primary interest 
rather than secondary. 

The concept of a team approach to software development has been widely 
discussed in the literature. This methodology offers a team approach 
that may by necessity draw the team members from various organizations. 

3.U.1 Independent Requirements Analysis 
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The review and analysis of the requirements specified at both the functional 
level and the detailed design level by a group of one or more analysts who 
did not participate in the requirements definition or generation is 
recommended. 

The review and analysis of specifications are critical functions. Initial 
specifications often contain ambiguities and are not always complete. 

The independent analysts review the specification, talk to the requesting 
organization to determine if the specification accurately states the 
requirements, talk to the implementers (in-house or contractor personnel) 
to determine if their interpretation of the requirements is the same as 
the requestor's interpretation, and act as coordinators to resolve any 
inadequacies and conflicts in the requirements. 


d 


H 

IfelMtl 


n 


Prompt feedback must have a high priority to assure that design changes 
can be made at the earliest possible stage of development. 

The independent analysts review and analyze the specifications for 


. useability 
completeness 
. clarity 
. continuity 
. uniformity 
. traceability 
. testability 

The results of the analysis are presented to the approval authority, 
at preliminary design reviews and critical design reviews. 

Requirements Analysis and Feedback is discussed in detail in Section U.l. 


.ri! 

.»!! 
" 1 


Mi 




rrn 
-:!i 
: ■ 







3-lU 




3. h.2 Independent Test and Evaluation 

The design and performance of test and evaluation functions "by an independent 
organization is recommended. 

For a new system* the test engineers review the requirements of the 
system and design asserted test criteria with accompanying test cases 
that will demonstrate that the implemented system performs as specified. 

In the case of existing systems the independent test engineers review 
the improvements and corrections to he incorporated in the modif icatiojis * 
and design asserted test criteria with accompanying test cases that will 
demonstrate that the modifications cause the system to perform according 
to the requirements as well as demonstrate that the unmodified parts of 
the system are not degraded. 

Asserted test criteria are doctimented in the test plan while also being 
placed within the respective programs using the embedded assertion 
language techniques described in Appendix C. 

The test cases are executed on an informal basis until the fonnal mod is 
made, the discrepancies are recorded and given to the designers/programmers 
for correction, the test data is evaluated to assure that the test 
objectives are being met, and a formal test of the system is made. A 
final test report 'is prepared attesting to the extent to which the require- 
ments are met 'as demonstrated by the test effort. 

,‘i 

If certification is required, the test report is the basis for certification 
by the independent test and evaluation group. 

Testing, evaluation and certification are discussed in Section k,3 and 

4. U. 


3.4.3 Team Approach to Programming 

In formulating a methodologjr for producing reliable software serious consid- 
eration should be given to the use of a team approach. Rather than recommend- 
ing a specific approach, however, it is suggested that teams be tailored to 
the size of a project and the operational environment available at the 
developing location. A degree of flexibility should be provided in estab- 
lishing the exact make up of a programming team. Past experience has 
shorn that blind adherence to a reportedly "ideal" team organization can 
be counter productive. Various team approaches have been advertised 
widely in the literature. A discussion of several of these organizational 
schemes is contained in Appendix D. 


3.5 PROJECT DEVELOPMENT PLAN 


The development of a brand new system or the incorporation of a major 
function into an existing system needs to be carefully planned. The 
development plan should establish the goals of the project in terms of 
its purpose, and its function. It should define the methodology to be 
used for the implementations, and the controls to which the development 
is to be subjected. 

3.5*1 Establishing Goals 


In considering the development of a system with quality considerations 
built in, it is necessary to understand the scope and purpose of the 
total system at the very beginning. The top-down development of the 
system structure allows early visibility of the entire system, and a 
more accurate assessment of realistic performance and scheduling criteria 
at the outset. 

The performance criteria are stated as requirement sets that satisfy the 
purpose of the system. The scheduling criteria are stated as milestone 
sets that satisfy delivery requirements. In general, the lack of early 
quality-producing considerations will adversely impact the schedule, 
since potential problems at the beginning become real problems later on, 
requiring correction time that might have been avoided. 

Therefore, the goals to be established are those that will cumulatively 
result in a reliable system. Some of the major goals are: 

1. produce a complete and consistent preliminary design that contains^: 

. major software functions which either directly correspond 
to or can be easily traced to explicit software requirements. 

. a set of test criteria which can be used in the formulation 
of a comprehensive test plan for validating and verifying 
the completed system. 

. a complete picture of the overall structure of the software 
system. 

. enough detail I'egarding the allocation of functional 
processing requirements to designated software elements 
to support a thorough and credible demonstration of design 
feasibility and validity. 

2. produce a detailed design that: 

. is an extenfiion of the top-down design concepts incorporated 
into the preliminary design. 

. expands the test criteria to be associated with the validation 
and verification of the resulting system. 


3-i6 


. provides for a supervisory control routine for the 
implementation of each functional capability® 

. provides clear traceability back through the preliminary 
design to the requirements and forward into the code 
and as-huilt documentation. 

3. define and systematically carry out a series of reviews 
designed to: 

. create mutual understanding of the requirements by 
the requestor, the designer, the implementer , and the 
tester (design reviews). 

. communicate the existence, evaluation and correction 
of ’problems or potential problems (CCB). 

. communicate the configuration status of the developing 
system in relation to its specified configuration (CCB). 

confirm that the code that implements the system 

will perform the functions required of it (walk through). 

. assess the extent to which the tested system meets the 

requirements specified for it (formal qualification review). 

4. provide the designers, programmers, and testers with the aids 
(in the form of tools, guidelines and standards) that will help 
them to achieve and verify quality inthe product. Of particular 
importance is the timely availahility of these aids. 

5 . Establish and maintain the controls that will preserve the 
integrity of the system at all stages of its development. 

6. Motivate the personnel by: 

. providing them with opportunities to be flexible and 
innovative within the hounds of the controls placed 
upon the software development. 

. involving them directly in the scheduling of their own work 
as part of the Software Development ITotebook concept. 

There are a nxunber of other goals that could he defined, but they can he 
classified into subsets of the above goals. For instance, the subject of 
documentation is not directly addressed, but is defined by the standards 
in item 3j and is procedurized in the controls in item 4. 

In addition, the budget and milestone goals are not addressed, as their 
net effect on reliability generally comes about indirectly by causing 
pressure if estimates were not meaningful, or by avoiding pressures if 
they were. 
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3.5.2 Selectiog and Use of Aids 


The selection of proper aids to be used during design. Implementation and 
testing is critical to achieving and maintaining quality in large systems. 

Even more crucial is their availability at the time they are to be used. 
Availability of tools and techniques includes adequate procedures for 
their use. Tools that must he developed must be designed early enough 
to be built and thoroughly tested before they are used. Tools that are 
procured must be installed and cheeJced out in the environment in 'which 
they are to be used. 

The size and complexity of the system being developed, the hardware /soft- 
ware support resources available, and the tradeoff of value vs. cost 
are all factors to be considered in the selection. 

Some tools and techniques can be advantageously applied to almost any 
systeiti. For example, the walk-through technique is valuable even on 
small ” one-time-only" programs, the difference being a matter of degree. 

For the more trivial programs, an hour spent by another programmer 
informally reading the code may save several debugging runs, while on a 
large complex system public presentation of the code may be the best 
approach. Both are applications of the walk-through. 

Obviously, some tools must be selected to appropriately fit the environment. 

A machine or language-dependent tool must be customized, and even a 
portable tool is more often than not "almost" portable. 

Probably the most important aid and the first one that should be available 
is the programmer's reference manual, A comprehensive reference manual 
containing "what every programmer should know" about the resoui'ces avail- 
able to work with, the environment he mus't work within, the programming and 
documentation standards he must comply with, and the guidelines for developing 
good, standardized code. This manual is the authoritative sotirce defining 
the ground rules and conditions for developing the software. 

• Other tools and techniques which are candidates for selection are: 

, PORTRAK Structuring preprocessor 
. Documentation aids 
- Checkout /debugging aids 
, Updating aids 
. Walk-throughs 
, Desk Checking 
. Standards Checking Tools 
. Execution Analyzers 
. Path Analyzers 
. Cross Reference Analyzers 
Test Data Generators 
. Test Case Selectors 
. Report Writers 
. Performance Analyzers 




Individual tools are examined and their capabilities are presented in 
Appendix A and Appendix C of the Final Report for this Study. The 
appropriate point at which to apply these tools and techniques is dis- 
cussed in Section 3. ^.3. 

In the on-going development of large systems that take several years to 
complete, the evaluation and selection of aids should he performed on a 
periodic basis in order to take advantage of new technologies or to 
supplement existing technologies with enhancements. 

The GSFC computer configuration (the IBM series with TSO capability) 
coupled with the use of FORTHAIf as the universal programming language 
lends itself to the use of several general purpose tools that are 
either portable or easily adaptable to the con^juter center environment. 
These tools are application independent, therefore scheduling of their 
installation to meet a development schedule is not a prime consideration. 


Other tools that show promise are either designed for a different environ- 
ment (i.e. , language and/or machine) or are application-dependent. 

These tools should be considered as candidates for development specifically 
for use at GSFC. 

3-5*3 When to Do What 


The importance of timing is paramount in the development of software using 
the built-in quality concept. The decision to use certain techniques 
and tools at a point that allows errors and faults to be prevented 
rather than detected must he made very early in the development cycle. 

The following set of charts show the points at which various quality 
considerations should he imposed. The development cycle is shown as 
the conventional series of steps shown in Figure 3.1. While actual 
development does not take place in such clearly delineated steps, the 
phases serve to show the framework upon which decisions can he imposed. 
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Figure 3.1 (continued) 
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Figure 3.1 (continued) 
EFFECT OF EARLY QUALITY CONSIDERATIONS 
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Section 4 

SOFTWARE VERIPICATIOH/VALIDATION TECHNIQUES 


U.l GENERAL 

The verification and validation of software is the most difficult process 
of software development, and therefore potentially the most costly. The 
coat depends largely upon the emtent of reliahility required of the 
software. However, the application of some specific tools and techniqies 
can increase the probability of detecting errors, reduce the time required 
to detect and remove them, and help detect them at the earliest possible 
point in the development cycle. This can effectively reduce the cost of 
achieving a required level of reliability. The verification/validation 
process begins with the analysis and verification of initial requirements 
and continues throughout the entire development cycle and into the operational 
state of the system. 

This section deals with five aspects of verification/validation: 

1. Requirements Analysis and Feedback 

2. Code Analysis and Verification 

3. Program Validation 

k. Program Certification 
5, Reliability Determination 

The analysis of specifications and the formal and informal reviews of these 
analyses are discussed in relation to NASA Goddard's current procedures. 

Code analysis and verification techniques and tools are availshle and can, 
in many instances, be applied dirs'^tly to the NASA Goddard environment. 

These include both manual and automated methods of analyzing and verifying 
the code. 

Testing of the software is done in several phases. Each is discussed, 
with recommendation as to tools and techniques available to enhance the 
process. 

VJhile certification is not a direct requirement of NASA Goddard, its 
performance is discussed in relation to its positive effect on software 
reliability. 

The need for assessing the reliability of the software required techniques 
that are still in an experimental state. However, achieving a reasonably 
accurate determination is possible. The most promising methods of 
determining reliability are offered for consideration. 
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4,2 EEQUIEEMEWTS MALYSIS MD FEEDBACK 

The most impcrtant aspect of designing systems with built-in quality is 
the early verification of requirements. Analysis of several large 
systems shows that as requirements analysis and design time increases, 
testing time decreases. 

In general, a high percentage of errors are attributable to conceptual 
errors. The early detection of these errors reduces or eliminates their 
impact in later stages of development. 

This section addresses the problem of assuring that the requirements 
are adequately defined and stated, and that the design reflects the 
requirements correctly. 

This methodology recommends a series of reviews to assure that the intent 
and the requirements are compatible and that the requestor, the designer 
the implementer and the tester mutually understand them. 

All analysis and review activities should whenever possible be performed 
by an independent analysis group which may be either an internal or outside- 
contractor group of analysts. 

4.2.1 System Bequjrements Review (SRR) 

The purpose of the System Requirements Review (SRR) is to assure that the 
system requirements are feasible and that they are complete and unambiguous. 
The review may include the results of: 

. mission analysis 
. simulations of the system 
. functional flow analysis 
. preliminary requirements allocation 
. system/cost effectiveness analysis 
. trade-off studies 

. integrated logistic support analysis 
system interface studies 
. program risk analysis 
. producibility analysis 

. technical performance measurement planning 
integrated test plan 
. data management plan 
. configuration management plan 
. engineering integration plan 
, acceptance criteria generation 
. system safety definition 

Special attention is given to: 

risk factors, their identification and ranking as pointed up in 
the system/cost effectiveness analysis and technical performance 
measurement plan analysis, their avoidance/reduction and control 
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as indicated by analysis of trade-off studies, test planning, 
hardware proofing, and technical performance measurement. 

. significant trade-offs between stated system specification 

req.uirements /constraints and resulting engineering requirements/ 
constraints. 

. significant producibility considerations that are visible early 
in the program, such as manpower loading and hardware availability. 

For large systems, SRR's may be conducted for each operational and support 
subsystem depending on the natixre and complexity of the program. 

4.2.2 System Design Review (SDR) 

The objective of this review is to evaluate the completeness, traceability, 
correlation, optimization and the risk associated with the proposed 
system design. It encompasses the total systems requirements, i.e. , 
operations /maintenance/test, /computer programs /facilities /personnel/ and 
procedures. A su m mary review of the items covered in the System 
Requirements Review that produced the above definitions is included. 

The end resudt of the review is the assurance of a mutual technical 
understanding of the validity of the system specification and the 
engineering/cost realism involved in producing the system. The 
following items are to be achieved in the SDR. 

1. Insure that the updated/completed system specification is 
adequate and cost effective in satisfying validated 
mission requirements. 

2. Insure that the allocated requirements represent a complete 
and optimal synthesis of the system requirements. 

3. Insure that the technical program risks are identified, ranked, 
avoided, and reduced through; 

a. adequate trade-offs 

b. a responsive test program 

c. subsystem/component hardware proofing 

d. implementation of comprehensive engineering 
disciplines such as worst case analysis, failure mode 
and effects analysis, reliability analysis, and 

st andardi zat i on . 

4. Identify how the final combinations of operations, maintenance, and 
tests and acceptability requirements have affected overall 
program concepts. 

Insure that a technical Tinderstanding of the requirements has been 
reached and technical direction is provided to the implementers . 
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The SDR re-addresses the items reviewed in the SRR, plus the following 
items : 

1. updated design requirements for operations /maintenance functions. 

2. updated operations /maintenance requirements for facilities. 

3. updated requirments for operations /maintenance personnel 
and training. 

k. evaluation of 

a. system design feasibility and system/cost effectiveness 

h. capability of the selected hardware/ software configuration 
to meet the requirements of the system specifications 

c. allocated inter- and intra- sy-.Lem uterface requirements 

d. specific design concepts that ma^ require development toward 
advancing the state-of-the-art. 

e. the ability of requirements items to meet overall system 
requirements and compatibility between requirements 
items and configuration item interfaces. 

f. reliability trade studies 

g. review of the specification of critical items to assirre 
their traceability/correlation to the validated mission/ 
support requirements. 

h. review of all available test documentation, including 
subsystem and system test plans to assure that the 
proposed test program satisfies the test requirements 
specified in the system specification. 

i. review of computer programming requirements including 

. type of processing, such as on-line processing 
off-line processing, parallel or multi-processing, 
multiprogramming, time sharing, etc. 

. a gross description of the size and operating 
characteristics of all computer programs, 
including data bases. 

. a description of the requirements for system exer- 
cising and Identification of functional requirements 
{exercise configuration, conditions, missions, 
frequencies, functional simulation, recording and 
analysis) and identification of major elements 
to implement the exercising capability. 
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. identification of programs required throughout 
the system, such as operational programs, 
diagnostic programs, test /debug programs, 
simulation programs, exercise and analysis 
programs and other support programs. 

. identification of computer facility resources 
needed to support the developing and operational 
system. 

4.2.3 Preliminary Design Review (PDR) 

The preliminary design review is a formal technical review of the basic 
design approach. For large programs, a PDR is conducted for each 
functionally related group of configuration items. The PDR is the 
most critical review of the software development review series. This 
is the point where the conceptual design is accepted and the software 
system is built upon it. Errors left imdetected in the design are 
often propagated throughout the other phases, causing grief in the 
later stages. 

The items reviewed in a software PDR include; 

1. Computer program functional flow. 

This information is completed to the level of flow charting 
which identifies the allocation of computer program components 
to functions and depicts the sequence of operation within 
the system functional flow. 

2. Storage Allocation Charts, describing the manner in which 
available storage is allocated to individual computer 
programs. Timing, sequencing requirements, and relevant 
equipment constraints used in determining the allocation 
are included, 

3. Control Functions Description containing a description of the 
executive control and start /recovery features for the computer 
program system. It includes the method of initiating system 
operation and features enabling recovery from system malfunction 

4. Structvire and organization of the Data Base identifying data 
types and characteristics, structure and layout, and allocation 
of data storage. 

5. Standards and conventions to he used in generating and testing 
the system. 

6. Test plans in relation to their ability to demonstrate that 
the completed software system satisfies the requirements. 


T. Configuration identification of major modules,' 



8. Interface definitions describing the relationships of software 
system components , for assurance that a particular item does 
not adversely iiipact or is not adversely impacted by other 
system elements. 

PDR's are conducted until the software system design is accepted as 
satisfactory. Uo detailed design or coding is performed until the 
preliminary design is complete. 

Critical Design Review (CDR) ^ 

The purpose of the Critical Design Review is to determine that the detail 
design of the configuration item under review satisfies the design require- 
ments in the specification for the item, and to establish the exact inter- 
face relationships between the configuration item and other related items . 

The CDR for each configuration item is conducted prior to the release of 
the design for production of the software, and the result of each CDR is 
to commit the design to production. 

For computer program configuration items, the CDR is a formal technical 
review of the item design. The CDR is normally accomplished for the 
purpose of establishing integrity of computer program design at the 
level of flow charts or computer program logical design prior to coding 
and testing. When a given item is a complex aggregate of computer program 
components, the CDR is accomplished in increments during the development 
process corresponding to periods at which components or groups of components 
reach the completion of logical design. For less complex items, the CDR 
is accomplished at a single review meeting. 

The primary product of the CDR is formal identification of specific computer 
programming documentation which will be released for coding and testing. 

Documents to be reviewed include; 

, Draft of a complete detail design specification for the computer 
program configuration item ^lnder review. 

. Supporting documentation describing results of analyses, testing, 
etc. 

. Documentation of allocated resources for the item 

. Test requirements for the item including asserted program properties 
(reviewed for completeness and technical adequacy) . 

. Test documentation required to support the test requirements, test 
procedures in particular. 

, Configuration documentation for each item. 
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4.3 CODE ANALYSIS AND VERI?ICATlON 


Attempts to check code for accuracy and efficiency have taken many forms. 
Two maniial techniques have been found effective in reducing errors when 
applied systematically. There has also been a proliferation of tools* 
developed to a large extent on an experimental haslsj that are designed 
to enforce standardization of code and to aid in checking it for incon- 
sistencies, incompleteness, and other structural faults that may cause 
problems later in the execution of the logic. They also help in deter- 
mining efficiency of the code in many cases. 

Except for the manual technique of "walk throughs", no currently available 
tools address the function of a program. Some promising steps are now- 
being taken to address function in application-independent tools. One 
such technique involving the use of an embedded assertion language -with 
accompanying tools is presented in Appendix C- 

Tools that are application-dependent and/or environment-dependent must 
be designed and built as needed. 

4.3.1 Manual Techniques 

The following two manual code checking techniques should be performed as 
standard procedure; 

a. At the completion of the coding of any module, and prior to 
submittal for compilation, the application programmer shall; 

(1) desk-check his module, following the procedures described 
in 4, 3. 1.1 until no additional errors are discovered; 

(2) update the flowchart of the module to reflect asy coding 
modifications; (3) review the module's flowchart with his 
auxiliairy programmer; (4) submit the module for desk-checking 
by the auxiliary programmer ; (5) repeat the above steps (1-4) 
until no additional errors are discovered; (6) obtain the 
auxiliary programmer's approve! on. tbe module development form 
(MDF - see 4. 3. 1.2). 

b. Obtain an error-free program compilation 

c. Update the program flowchart to reflect the valid compilation. 

d. Review the updated flowchart with the auxiliary programmer 
and obtain his approval on the RDF. 

e. Prepare sufficient modiile development test data, as described 
in 4.4.1. 

f. Submit the module program design language (PDL), description or 
flowchart to a group walk-through, as defined in section 4. 3. 1.2. 

g. Test the module with the test-data; review the results of each 
test run with the auxiliary programmer. 
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h. After development testing has been satisfactorily completed, 

a public presentation of the code -will be conducted (see i}. 3.1.2). 

^,3.1.1 Desk Checking Procedures 

The sample procedure listed below illustrates a method which can be followed 
in desk-checking a FOETRAIJ modtile: 

1. Answer the following checklist questions : 

a. Does the commentary block define the purpose, names and 
definitions of all variables that are transmitted to, 
and/or from, the routine; and contain version date, 
programmer's name, references and any special considerations? 

b. Does the commentary block immediately follow the subroutine 
name? 

c. Are there sufficient comments interspersed throughout the 
code to explain the general logic flow? 

d. Have embedded assertions been included both as text in the 
design documentation and as comments in the code for checking; 

data integrity 
entry/exit constraints 
result validity 
local data constraints 
local addressing constraints 

e. Are declaravives in the following order? 

TYPE statements 
DIMEHSIOK statements 
BLANK COMMON statements 
Labelled C01-5M0N statements 
EQUIVALENCE statements 
DATA statements 

f. Are the declsiratives blocked so they are easily readable? 

g. Do all floating point variables begin with letters A-H or 0-Z? 

h. Do all fixed point variables begin with letters I-N? 

i. Do logical and complex variables begin with letters appropriate 
to the function of the variable? 

j. Do all 'variable and subroutine names suggest their function? 

k. Do all variables in a Common block use the same name in 
every subroutine in which it appears? 
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l. Are varialDles passed "between sularoutines "by the use of 
COMMON rather than hy calling parameters? 

m. Do all COMMON "blocks contain less than seven arguments? 

n. Has EQUI"VALENCE been used to identify specific locations 
in COMMON block arrays? 

o. Are all RETURN statements, C-OTO statements and CALLS 
and function calls Itilielled? 

p. Is the normal RETURN statement labelled 999’ 

q. Are all exceptional RETURN statements labelled with a 99^? 

r. Are all labels in ascending order? 

s. Are all COMMON variables initialized in a BLOCK DATA 
subroutine, and defined by COMMENT cards? 

t. Are the variables in a COMMON block in the following length 
order? 

. complex*i6 

. COMPLEXES 
. REAL*8 
. REAL*U 
. LOGICAL 
. INTEGER»2 
. INTEGER* 2 
. LOGICAL*! 

u. Are all continuation cards numbered in sequence in column 6? 

V. Do parentheses balance? Start from the left with 0 and add 
1 for each parenthesis and subtract 1 for each right 
parenthesis. The count should never become negative. 

If parentheses balance, the count will end up to 0. 

However, this does not indicate correct grouping. 

w. Do FORMAT statements follow the declaratives and precede 
the executable code? 

X. Are all FORMAT statements labelled in the 99xx range in 
ascending order? 

y. Do the subroutines have less than 100 lines of code? 
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2 . Using the Program Design Language (PDL) description or subroutine 
flowchart, manually follow the execution sequence of routine 
logic. This entails: 

a. Preparing sufficient test data to inspire that each function 
within the routine will be exercised at least once; 

b. Manually record each change of program data, variables, counters, 
and indexes {using the prepared test data to drive the 
routine) ; 

c. Verily that the program logic flow accurately reflects 
the program requirements; 

d. Correct all discovered errors and repeat the above process. 

The desk check procedures can be greatly assisted by using a standards 
checking tool. This type of tool can check for standards violations and 
flag them for correction. One such tool is described in more detail in 
Section 4. 3. 2.1. 

4. 3.1.2 Walk-throughs 

Management shall divide the programming staff into groups of three or more 
programmer /analysts. Bach group constitutes a review group, which will 
collectively review each group member’s programs. There should be two 
reviews during the development of each program developed by a group member: 

(l) prior to development tes’t, but after coding is complete; (2) subsequent 
to development testing. These reviews are conducted to ferret out program 
logic errors and to insure that the program has been thoroughly tested. 

Each review is termed a "walk-through" , where the application programmer 
conducting the review explains his module to the other group members. 

These reviews typically take the fom of a viewgraph presentation of the 
modules PDL or flowchart, where the cognizant programmer traces the 
logic flow (i.e., wlaking the other group members through the module 
logic). The initial review is informal and made to programmers. The 
public presentation after development test is formal and is made to 
the Design Group. 

During each review, errors may be detected by the members of the review 
group. Each error discovered will be recorded by the auxiliary programmer 
and serve as an action item list for the cognizant programmer. During 
both reviews, the action items discovered will be recorded in the Module 
Development Form (MDP) (see Figure 4-1); prior to final approval of the 
development testing completion for a particular routine, the review group 
should insure that all action items have been corrected. 
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MODULE HAME 


HEAD PROGRAI-ffidER 


BACK-UP PROGRAMMER 


DESK CHECK DESCRIPTION 


DATE CODING 
COMPLETED 



DEVELOPMENT TEST DESCRIPTION 


DATE DESK 
CHECK COMPLETED 



1 i 


WALK-THROUGH APPROVAL 


DATE WALK- 
THROUGH APPROVED 




DATE DEV. 

TEST COMPLETED 




I WALK-THROUGH APPROVAL 

1 

1 

DATE WAIiK- 
THROUGH 






Figure 4-1 

SAMPLE MODULE DEVELOPMENT FORM 
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I;. 3. 2 Automated. Teclm.ique5 


ii,3.2.1 Standards Checkers 

The use of a standards checking tool can he used to comp-lement the manual 
desk checking techniques mentioned earlier. Two tools seem worthy of 
mentioning as examples. The first, PFORT Verifier, Bell Laboratories , 

Murray Hill, New Jersey is a very useful tool for checking the portability 
of FORTRAN programs. The second. Standards Auditor, Computer Software 
Analysts, Inc., Los Angeles, California is a tool which was originally 
built to check the coding standards for the Arry's Site Defense Project 
being built by MDAC and TRW. 

The PFORT Verifier checks a FORTRAN source program, for adherence to a 
portable subset of MS FORTRAN. Subprogram communication is checked 
through common and argument lists . Debugging and documentation aids 
include subprogram cross reference giving type, usage and attributes 
of each identifier with a list of statements in which it occurs. Also 
provided is a summary by subprogram listing argument attributes , common 
blocks used, subprograms called, and the calling subprograms. PFORT 
has been installed at a number of locations and is available from Bell 
Labs. Appendix A contains sample output from the PFORT system. 

Standards Auditor currently checks 38 coding standards. It has a suppression 
capability that allows selection of any subset of these 38 standards. 
Additional standards can he readily added. 

It has been found that the most benefits a'^crue from cheeking a small 
core of standards which include statement location, comments and module 
size. Standards Auditor is marketed as a program product by CSA. A 
program was supplied to CSA for analysis in connection with this study, 
however, no output was received for inclusion in this study report. 

If. 3.2,2 Execution Analyzers 

There are many execution analyzers that have been built on both a commercial 
and e:!q)erimental basis. Most are designed for FORTRAN code, with a 
few commercial ones handling COBOL code. 

The Boole and Babbage problem program evaluator (PPE) already in use at 
GSFC, is language independent since it is applied to object code. Tools 
such as the McDonnell Dougj.as Program Evaluator and Tester (PET) , the 
CAPEX FORTUNE, and the National Bureau of Standards Analyzer instrument 
the FORTRM source program. 

PPE is a valuable tool for measuring the performance of an executing 
program. Its main advantage is that it resides in the same region as 
the problem program being measured and can readily be applied to programs 
during production rTms. This helps to determine the performance of a 
program in a real use environment while operating with actual rather than 
test data. The executing program is not modified in any way, therefore, 
there is little chance of degrading the program's functional capability, 
except where a timing function is involved. 
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The disadvantage inherent in the use of PPE is the difficulty encountered 
in interpreting the results. 

For a total picture of a program’s efficiency, PPE is a good tool to use. 
While other tools can detail the frequency of execution of each statement , 
timing considerations are difficult to arrive at accurately. 

A second tool is recommended jo supplemeht PPE at GSFC. This is the FORTEAN 
analyzer, PET, produced hy Mclonnell Douglas. 

s tool is designed aroujicl a preprocessor/postprocessor organization. The 
preprocessor inserts softv/are prohes into the target code. The postprocessor 
analyzes the data collected hy the prohes, and -writes several summary 
reports containing the resialts. 

Run time statistics includs: 

1. the number and percentage of the total of executable statements, 
non-executahle statements, and comments. 

2. the number of and percentage of all potential executable 
statements that were executed one or more times. 

3. the number of and percentage of program branches tested. 

h. the number of times each branch was executed. This includes 
branch counts for logical and arithmetic IP conditions , 
plus coB^juted and assigned GOTO's branching histories. 

5. the number and percentage of subroutine calls that were executed, 

6. the number of times each subroutine was called, and the names 
of those subroutines that were never entered. 

7. relative timing for subroutine executions 

8. the number of times each executable statement was executed. 

9. the TTiin-imiim and maxi mum values attained by an assignment 
statement variable or DO loop parameter. 

10. the first and last values attained by an assignment statement 
variable or DO loop parameter. 

The data collected and reported by PET can be used to: 

. show areas of high activity during execution of various test 
cases. 

. show untested code 

. develop, test cases that exercise the entire program 
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While the application of PEI to the code does not prove the correctness 
of the algorithms, it does allow the observation of the behavior of the 
algorithms "with actual test data. Future plans include the incorporation 
of an assertion capability that will address the correctness of the 
algorithms to some extent (see Appendix C). 

The chief value of PET is its assistance in deriving adequate test data 
for development testing. By showing that all of the code was tested 
using valid test data, the credibility of a program's correct performance 
is enhanced prior to the final walk through. 

Other execution analyzers have been built for FORTRAN on the CPC and 
IMIVAC equipment, however, none of these tools are available on the 
IBM 360/370 series computers (see Appendices A and C) . 

^.3.2.3 Cross Reference Analyzer 

As programs increase in size, the problem of naming conflicts increases. 

This is particTolar3y true when enhancements are made to existing programs. 

A cross-reference analyzer creates glossaries verifying the consistency 
of symbol naming and usage. 

JOYCE is a tool produced by McDonnell Douglas which provides cross reference 
lists for FORTRAN programs. The symbols referenced include the names 
of any referenced module or functions, any entry points, and all I/O 
file references. The cross reference lists are also useful for finding 
typographical errors in coding and for cheeking a program’s logic flow. 
Sample outputs are contained in Appendix A. As is the case with many 
of the better tools examined in this survey, JOYCE is currently only 
operational on CDC machines. 

I+. 3 . 2.4 Path Analyzers 

There have been several attempts to develop a path analyzer that is easy 
to use and that is helpful in debugging and testing of code. However, 
the path analyzei's that are applied to source statements within a 
module are awkward to use and require that the user have intimate 
knowledge of the code and the function for which it was created. Their 
use will become more significant when they are used as forerunners to 
test data generators which are still in the eixperimental stage. 

Path analyzers at the subroutine level provide visibility over an entire 
program in regard to subroutine entries. 

The Automated Test Data Generator (ATDG) developed for RASA in Houston 
by TRW is discussed in Appendix A. 

ATDG is a path analyzer that is promising. However, it is written to 
run on a UNIVAC 1110 cos^uter. It is an interactive tool that requires 
a high degree of user involvement, and an intimate knowledge of the code. 

IL Is complicated to use and since it is used only at specific points 
in the development of a program, programmers often do not want to take 
the time to understand how it works. However, work continues in the 
simplification of its use. 
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When ATDG is completed, it may he a candidate for conversion to the 
IBM 360 configuration, particularly since it is already a NASA 
sponsored product. 

Another promising tool currently being developed by MDAC is DISSECT, a 
symbolic evaluation system, used to analyze progra m s -written in AUSI 
FORTRAN. 

When a program path is executed by running the program on a given input, 
the correctness of the path for that input can he determined hy examining 
the effects of the calculations carried out hy the path. If the path is 
executed "syrnholically" rather than td-th actual data, it may he possible 
to use a single execution to illustrate its correctness on a large subset 
of the input domain rather than on just a single value. Symbolic execution 
of a program is carried out by given dummy symbolic values rather than 
actual numeric (string, logical, etc.) values to all or some of the 
input variables of the program. An expression in the program involving 
variables with symbolic values is evaluated by substituting the current 
symbolic values of the variables into the expression. The resulting 
expression is then simplified algebraically. All operators having only 
actual as opposed to symbolic operands are evaluated in the normal way. 

The resulting expression is the symbolic value of the original expression. 

The command file is built for a DISSECT analysis of a subject program and 
is divided into a number of cases . The program is analyzed for each case. 

The system is used to examine desired program patts. 

A program path is a possible flow of control through the program. A 

path is feasible if at least one element of the program's input domain causes 

that path to he executed. In general, a complete set of DISSECT cases 

for a program should "cover" the program in some sense. One approach is 

to analyze each feasible path (up to some number of iterations of loops). 

Complex programs having many paths can he divided into segments £ ± 

analyzed using separate cases. 

A great deal of interesting research remains to he done in connection with 
using the DISSECT system to study tecbnlques for examining program 
correctness questions. Sample output is contained in Appendix A and an 
additional description of the system is presented in Appendix C. 

k,h PROGRAM VALIDATION 

The systematic validation of a computer program begins with the validation 
of its design, and in theory ends with the formal qualification test. 

However, in practice, validation extends into the maintenance phase and 
ends with the demise of the program. 

Therefore, this methodology addresses several kinds of testing that span 
the development pe:."iod and continues into the operational state. 

It may be argued that development testing is not really a validation process 
but it is included here as the testing that is the transition from the 
debugging to formalized validation efforts. Its importance is significant 
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■because it assures that the debugging process is complete "before the 
code is integrated into a system for testing. This reduces the chance 
of failures caused by st.'.’uctural errors, allowing concentration on the 
detection of logical errors during testing. 

The use of a formal qualification test against an exactly knoira configuration 
provides a baseline for the system. Any discrepant behavior of the system 
is documented. The decisions impacting system acceptance and disposition 
of discrepancies can be made in a formal manner with adequate and visible 
justification. The operational test provides a means of measuring the 
system's real time performance in the operational environment. This 
testing determines if the uptime requirements are met over a pre-specified 
period of time. 

Baseline tests exercise the critical functions of a system to determine 
the effect of changes. In particular, they are used to insure that the 
critical functions have not been degraded. Baseline tests and benchmark 
tests are considered synonomous in this context. 

4.U.1 Development Testing 

The purpose of the development test is to provide some assurance that a 
unit of code, generally a subroutine or group of subroutines that perform 
a function, works as it was intended. This is accomplished hy processing 
a trivial but realistic test case. 

The development test is designed and performed by the prograomier who wrote 
the code. He writes a test plan that describes the function of the code, 
the data he intends to use to demonstrate that code works properly, and 
the results he expects from the execution of the test . 

The programmer discusses his test plan w.’.th his auxiliary programmer 
and gets his concurrence. 

The development test should show that the program function is achieved, 
and also that all of the code is exercised hy the test data, so that an 
unusual occurrence of data combinations will not result in unhappy sur- 
prises later. The use of an execution analyzer such as PET will assist 
greatly in achieving these goals, by automatically checking program 
assertions and providing the statistics showing the behavior of the 
program during execution using candidate test data. 

The debugged routine(s) is linked to the basic system, the test case(s) 
executed, and the results documented. When the intended results 
are obtained, the program is ready for the formal walk through. 

The test plan, test data, and the documented results are placed in the 
software development notebook. The program is turned over to the 
validation group for formal validation testing. 

When development testing of new programs is complete, the code is placed 
under release control and entered into the controlled library. If the 
code consists of modifications to routines already under release control. 
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The test program must be planned early in the development cycle. System 
requirements must be analyzed for testability before the design of the 
system can be successfully completed. The formulation of a test plan 
must be based on a well stated requirements document. If a requirement is 
explicit, its testability can be readily ascertained. However, implicit 
requirements must be considered also, and the identification of these 
requirements and the determination of their testability often requires 
a great deal of though and conscious effort. 

The system test plan has as its goal the evaluation of the performance 
of the system. Its execution must provide positive answers to the 
question "Will the system do what it is supposed to do?" Since "what 
it is supposed to do" is specif’’' lly identified in the requirements 
docTiment, it is important to pla.e boundaries on the test plan to insure 
that all specified requirements have been Implemented, and that no 
unspecified requirements have heen implemented. 

The system requirements provide the foundation for the selection of 

test cases. Ultimately, the composite of all test cases, when successfully 

executed, verify that the requirements of the system have been satisfied. 

The process of test case selection begins with the groupings of requirements 
by function. An exan^le is shovm in Figure U-2. For a specific require- 
ment, objectives are defined and assertions f emulated which explicitly 
identify all the Implications of that requirement that are to be verified. 

As a result of specifying an objective, each requirement is clarified and 
any weaknesses and ambiguities can be identified and resolved. This 
process assures that each requirement is testable. 

Test design begins by considering each objective and answering the following 
questions regarding it: 

1. what are the outputs required to evaluate the perfomance? 

2 . what are the inputs? 




3. liow does tilae data need to "be analyzed to verify the objectives? 


what is the acceptance criteria upon which the pass/fail 
decision is based? 

5. what is the software/hardware configuration required to test 
the objective? 


The answers to these questions form the test requirements which when 
implemented in a test program, will verify the system performance. 

When all of the objectives have been identified, a grouping of the 
objectives is performed. The criteria for grouping may be predicated 
upon the commonality of the software/hardware configuration and the 
system inputs required to verify each objective. A group of objectives 
define a test case. 


At this point, the test cases are grouped together to form scenarios 
that provide the input for a test (see Figure ii-2) . 

Test cases are grouped on the basis of the operational or chronological 
relationships of the inputs. With the scenarios defined, the detailed 
procedures for the conduct of the test can be written. 

PROGRAM CERTIFICATION 

Certification of a system is the last step to he taken before acceptance 
of the system. The purpose of certification is to provide confidence that 
the system will work as expected with a specified degree of reliability. 

O 

The following definition of certification is given by R. C. White , and 
contains all of the elements that characterize certification. 

'^Certification is the act of author it at ively confirming that some 
set of characteristics are compliant with a particular set of 
requirements for these characteristics/capabilities. 

This act may he further characterized by the following features: 

. It is an official authoritative affimnation of the 
compliancy relation's existence. 

. It is issued by a recognized acceptable authority. 

It is consequent to an affirmative compliancy decision. 

. It may grant official acceptance. 

. It has such force as to encourage, if not compel, acceptance. 

. It has possible legal efficacy, as determined by the recognized 
authority and the source of his responsibility for certification. 














Construed as an act, certification is fundamentally "a single accomp- 
lishment complete in itself and essentially xmique", in contrast to 
the extended activity or range of activities, characteristic of 
compliance determination. Thus, it is an existence-confirming (of the 
compliancy relation) act, rather than the existence-determination 
activity of compliancy determination. The latter, to reiterate, is 
a prerequisite for certification and cannot, therefore, in the interests 
of consistency, he denoted by the term certification". 

The bulleted points in this definition are of particular interest to this 
methodology, since each represents the result of some quality-producing 
activity or activities already described in this document. 

The act of determining compliancy with the requirement set is the result of 
on-going analysis of the test data culminating in a final review, the 
Functional Qualification Review. The analysis and certification should be 
performed by an independent agency with recognized capabilities to authori- 
tatively confirm compliances . The acceptance of a system can then be 
based upon explicit recognition of its capabilities and any discrepant 
behavior that may be deemed non-critical and within the realm of acceptability. 

4 . 5.1 Compliancy Determination 

If the technique of defining test objectives for every requirements is used 
as described in Section 4.4 then it can he determined from the tests 
results whether or not each characteristic or capability of the system 
is compliant with a particular requirement of that characteristic or 
capability. 

f 

White maintains that the interpretation of compliancy is binary, i.e., 
the characteristic or capability either is or is not compliant. It 
cannot be partially compliant unless the requirement is decomposed into 
discrete suhrequirements which permit separate compliancy determinations. 
Therefore, the completed system can be certified as compliant if all 
requirements are satisfied, essentially forcing acceptance. However, if 
the system fails to confo 2 ma with one or more requirements , then a negative 
compliancy decision must be made and alternative action taken. This 
action may be to return the system for correction of problems, to 
accept the system without certification, to certify the subset of the 
system requirements for which conformity was established, or to modify 
or amend the requirements so that the compliancy decision is affirmative. 

The use of the test objective matrix described in Section 4.4 allows the 
evaluator to cheek every requirement or subrequirement for compliancy 
by the inspection, analysis and evaluation of the test results. Since 
ambiguous or incomplete requirements cause problems in defining test 
objectives and subsequently in determining compliance, the analysis and 
feedback of requirements in the initial stages of development not only 
helps create a cleaner design and implementation of the system, but 
pern^ts the development of a test plan that will provide a clear 
indication of requirement compliance. 
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Compliancy determination is an on-going activity that spans the test 
phase. The examination of requirements may hegin with the heginning 
of the test plan design and continues throughout. The compliancy 
determination for each requirement may he made as testing progresses for 
each subsystem. The final determination of compliancy for the entire 
system is ha 'd on the cumulative determinations for every requirement. 

The compliancy decision* either affirmative or negative, is a unique 
one-time process that is made at the Formal Qualification Review following 
the acceptance test or system test. 

4.5*2 Formal Qualification Review 

The objective of this review is to verify that the actual performance of 
the system complies with the requirements as specified. This verification 
is based upon an analysis and evaluation of the test results. 

The Formal Qualification Review exam?.nes the results of the analysis and 
evaluation of the test data with relation to the corresponding req'uirement 
set. The end result of the FQR is to deteraaine the disposition of the 
system ( acceptance /reject ion) based on the compliancy determination 
factors. All discrepancies found in the testing are presented, and a 
decision is made concerning their disposition, generally based on their 
criticality. 

The items presented are: 

1. the test objective matrix showing the direct relationship 
to the requirements and the extent to which the system is 
shown to comply with the requirements. 

2. the test plans and procediures 

3. a list of all successful functional tests 

4. a test report containing the anaJysis and evaluation of 
the test results. 

5. discrepancies detected during the acceptance or system test 

6. the functional and physical configuration audit data confirming 
the configuration of the system for which the test data 

is verified. 

7. a list of all completed manuals and handbooks to be used with 
the system. 

8. an affirmative or negative compliancy decision with recommendations 
for acceptance or rejection. 


U.6 RELIABILITY DETERMTHATIOE 


The ability to assess the reliability of the software has been a subject of 
much controversy. Many in the past have considered the attachment of any 
figure-of-merit to software as an impossible task. Recent research efforts 
have developed a number of interesuing models which have been applied with 
varying success to software. The key to applying these models involves the 
availability of error data corresponding to a given piece of software. 


Despite the fact that numerous individuals have recognized the need 
to save error data for some time, remarkably little can be said about 
software errors. A number of projects have introduced trouble reporting 
schemes and reams of paper have been generated, however, practically no 
analysis has been performed on the nature of these errors. Often the 
information requested on the trouble reporta is of little value for such 
analysis . > 

Based on the knowledge gained in developing a number of predictive software 
reliability models, McDonnell Douglas has designed a refined reporting form. 
After examination of a number of RASA and DOD reporting forms the following 
sample Software Malfunction Report (SMR) was produced (Figure ^-3). 


The software malfunction report (SMB) categorizes malfunctions into 
six general classes with specific malfunctions as a subset of the general 


' • 

The six 

general classes are 

1. 

"A” 

Arithmetic 

2, 

»B” 

Argument 

3. 

’•c" 

Logical 

h. 


Assignment 

5. 


System 

6. 

iipti 

Data 


Each general class contains varying numbers of specific error types. The 
specific error types were added to highlight the type of error in each 
general class . These specific error types are not fixed and they can 
be deleted or expanded. The specific error types may xmdergo several 
major changes until a large enough sample is obtained. 

With the availability of the error data by general class and the 
availability of timing statistics reflecting accumulated development 
testing times, software reliability models will gain increased accviracy. 

Assuming that software error data is gathered, one question still deserves 
addressing namely: how should the models be applied to the software 

error data? 
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ADVANCE INFORMATION SYSTEMS 
SOFTWARE MALFUNCTION REPORT 
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REPORT NO.-, 

PROGRAM NAME 

S 9 101112 13 1-q 15 16 171819 20 2122 23 

PROGRAM I.D. i ( t I I ( TT 1 MODULE LD, M I 1 T I I t I ORIGINATOR 

STAGE 2^ 25 26 27 

CHECKOUT Q TEST AND EVALUATION Q INTEGRATION Q USER Q 

SCOPE OF ERROR 28 23 30 31 32 

Specification □ design □ coding □ integration Q other Q 

general error type specific error 

arithmetic 3SQ I COMPUTATION; 2 OVERFLOW; 3 SIGN; 4 SCALING; 5 ROUNDING; 6 QUANTITY 
ARGUMENT 3?Q ^ FLAG; 2 CONDITION; 3 LOOP; 4 PARITY; 5 INDEX REGISTER; 6 INSTRUCTION 
logical 38 Qj 1 PROGRAM; 2 PARAMETER; 3 SEQUENCE; 4 COUNTER; 5 INCONSISTENCY; 6 CODE: 7 REQUIREMENT 
assignment 09 Q 1 ADDRESS; 2 ALLOCATION; 3SUBROUT1NE; 4 INTERRUPT; 5 INCOMPATIBLE; 6 ENABLE/DISABLE; 7 MOVE/SORT 
SYSTEM 40 Q , INTERMITTENT: 2 LINKAGE; 3 MASKING; 4 SYSTEM STRUCTURE; 5 TIMING; 6 PROCEDURE 


SEVERITY 

HIGH Q 33 

MEDIUM Q 34 
LOW Q 35 


12 3 4 5 6 

date 1 I I I i n 

CARO TYPE Q 


DATA 41 Q 1 STOHE/SAVE; 2 CONTROL CARO; 3 FORMAT; 4 CELL: 5 OUTPUT; 6 INPUT 


4243444546 4? 
NUMBER OF INSlfU'CnONS | 'f f DTI 


48 49 50 51 52 S3 54 

NUMBER OF INSTRUCTIONS FT' I '1 i f 'I DOES IT USE STANDARD y -5 fn 
FOR CORRECTIONS J » M ,1 ■ I SUBROUTINE ‘—I 



56 ^ 53 59 

IS module CALLASt E av ITSELF YES Q NoQ DOES IT USE OTHER MODULE YES Q NoQ 

606163 63^4 65 66 6769 *^ 7071 7 2 737470 75 77 

COMPUTER TIME TO PROGRAM HALT f ] I ! I ~T1 TOTAL TIME [ I f I I I I TOTAL TIME IN STAGE F j I M fl 



Figure 4-3 

SOFTWARE MALFimCTIOiJ REPORT 



As indicated in the Appendix B section on failure-rate models the initial 
testing of a program frequently does not corrc'spond to the underlying 
distributions characteristic of the ultimate, or steady-state testing 
conditions. While this is so, it is still necessary to record the errors 
found, whether or not the times of their occurence have any use in direct 
analysis. As indicated in the selection of the "zero” time for the 
models, there are good indications that an approximation to the total 
number of errors in a program can be formed on the basis of the total 
instruction count. When this coimt is known, an ”apriori" error-per- 
instruction factor can be applied and to form an approximate error content. 
This then provides a means of setting a realistic limit to what may be 
called the initial segment of testing. (In this way, the illustrative 
application the "zero” time was chosen on the basis of the estimated 
time of occurence of the half-total error] . 

In the absence of any apriori estimate of the total error content, for 
whatever reason it cannot be obtained, it is well to record the times 
of error occurrence. It is clearly of interest to establish a point in 
testing where the number of errors per time unit (per CPU second, or 
per calendar day) decreases. Generally when this occurs any of the 
models which have been described in Appendix B can be employed to 
obtain estimates of the total error count on the mean-time-to-f allure. 

In any of the models, this ultimate convergence insures that they can he 
applied without regard to the possible transient state of the testing from 
which the data is obtained. The parameter estimates so obtained are 
not as good in this case as compared to estimates obtained during the 
steady state, but they will generally provide good guidance nonetheless. 
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Appendix A 

AUTOMATED VERIFICATION TOOLS 
See Volume II 
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Appendix B 
SOyTtfAEE MODELING 


B.l SUMMARY 

Five detecf;ion (failure) rate models for the software error process are compared. 
The de-eutrophication model developed hy Jelinshi and Moranda has a failure rate 
which decreases hy a constant amiount upon the detection and removal of each error. 
In the geometric de-eutrophication model developed hy Moranda, the rate for the 
"next” error stands in a fixed fractional ratio to its prior value (geometric 
series). Iji the geometric-Foisson model, also developed by Moranda, the average 
number of errors found in a given time period stands in a fixed ratio to the 
average found in the preceding time period of equal length. The Shocman model 
assumes the detection rate of the (group of) errors following a debugging interval 
to be directly proportional to the remnant error content. The Schiek-Vfolverton 
model is a variation of the de-eutrophication process in which the detection rate 
starts at zero after each error and increases linearly until the next error with 
the slope of the line decreasing after each detection with the magnitude of the 
slope being directly proportional to the current error content. 

All models are based on the assumption that the detection rate depends on the 
number of errors remaining in the software package. 

In this task, for purposes of illustration and comparison, the models are all 
applied to the same data, consisting of a daily record of the number of errors 
found in the debugging of a program and the CPU time used. Maximum likelihood 
estimates of the total error content, mean time to next error, and the degree 
of testing completeness are developed from a small time segment of the date and 
estimates are compared where possible. 

The estimates of total error content are compared to the total number eventually 
found. Estimates of the MTEF are developed for local and remote time periods. 

Finally, the variances /covariances of three of the models are developed. 

B.2 DESCRIPTION OF MODELS 

B.2.1 De-Eutrophication Process (Descript 5.on) 

This model, developed by Jelinski and Moranda of MDAC, is based on the assumption 
that the rate of detection of software anomalies (or errors) is proportional, 
at any time, to the current error content in the software package, and that all 
yeninant errors are equally likely to occur. This model is illustrated in Figure 1. 
The initial detection rate is given by NtJ>, where N is the Initial error content, 
and is the proportionality constant, but which clearly represenbs one "error’s 
worth" of contribution to the hazard or detection rate. 
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Figure 1. De~Eatrophication Process 


A typical realization of such a process is depicted in Figure 2, where errors 
are indicated hy the S-functions shown, and the time between errors, which is, 
in reality, random, is purposely indicated here as increasing steadily. 


INCIDENCE OF 
ERROR 






Figure 2. Typical Realization. of the De-Eutrophication Process 


The data for analysis consists of the sequence of times between errors: 

•• The development of maximum likelihood estimates for the two para- 

meters shown SKplieitly in Figure 1 is made in Section B. 6.1 .The essential facts 
involved in this development are; the xmiform or constant conditional failr’re 
rate for the ith error implies an exponential distribution for the associated 
time, with partaneter, {U-i+l)tf>; and the 3^'s are statistically independent. 

The application of the maximum likelihood technique produces the two equations: 
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where T is the total time ^ X, 

• -I 3 
1=1 


Applications of this model hav© been made to data sets obtained during the 
development of two large-scale real-time systems; one, the Navy Tactical Data 
System and, the other an Apollo-related software package. These were reported 
first in the original paper (Reference l); subsequently, updated information 
was obtained. This new data permitted a comparison between the predictions, 
which had previously been made and the realised data, in the form of Trouble 
Reports generated during the development of the Naval Tactical Data System 
during the forecast time period. The comparisons (three modules) are con- 
tained in a second report [Reference 2] . Those comparisons showed a remarkable 
consistency between the predictions and the realizations. In those applications, 
time was measured in units of days (calendar). 

B.2.2 Geometric De-Eutrophication Process (Description ) 

A variation of the de-eutrophication process has been found useful. In this 
form the detection rate decreases in a geometric progression on the occxirrence 
of each individual error; the times between errors are random instead of fixed, 
the errors are treated individually instead of by groups. This is shown in 
Figure 3. 



Figure 3. Geometric De-Eutrophication Process 
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The mathematical analysis for this model is given in Reference 2. The develop- 
ment parallels that used in the original paper: the are exponentially dis- 

tributed with parameter Rki”!, the observations are independent and the likeli- 
hood function is therefore the product of exponentials. Maximizing the logarithm 
of the likelihood produces the two equations: 


and 




2 



(3) 


ik) 


where k and D are described by Figure (3) and n is the number of intervals used. 
All sums are over the range 1 to n. 

B.2.3 Geometric Poisson Model (Description) 

The Geometric -Poisson Model is described by Moranda in Reference 2. The model 
is shown in Figure 4 . As indicated, the data is assumed to be reported only 
periodically {by week or month). The detection rate is shown to decrease in a 
geometric progression; each time interval, T, has a rate which is a fraction k 
tames the previous interval’ s rate (0<k<l), and represents the Poisson parameter 
for the initial collection interval. 



Figure 4 . Geometric -Poisson Model 
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Shocauan Model (Description aod Critique) 

The model described by Shooman initially in November 1971 was presented 
in improved form in January 1972 in a japer Jointly authored by 
J, C. ‘Dickson, J, L, Hesse, A* C, Kuenta and M, Shooman [References 3 and U], 
The same model and results were described in 1973 and 1975 by Shooman 
[References and 5]» 

In the Shooman development, the model is discussed in terms of the 
factor 

Y = K[% » 

^T 

where; K is a constant "which can be estimated by the ratio of the number 
of catastrophic errors detected to the total numbers of errors detected"; 

Ej is the total number of errors; is the total number of instructions, 
and "normalized" number of errors which have been corrected 

up to the (total) debugging time t; and rp is the instruction processing 
rate. 

As will be later shown, there are several subtle points which must be 
resolved before the Shooman model can be obtained. It is sufficient 
here to note that the instruction count Iip does not enter into the 
problem since the "normalization" required to form eliminates it. 

The parameter t, which represents debugging time, does not enter the 
analysis, nor do rp occur as a separate factor, it is obviously insepar- 
able from the factor K. 

Thus, in its essence, the Shooman model sets up a direct proportion 
between the detection rate y and the ciu'rent error content. 

In Shooman* s original paper (Reference 3) when he sets out to find the 
unknowns, ©j and K, the "constant error rate model" is employed, the 
assumption being that the 

= Po" 

To quote Shooman, the value of p_ is "evaluated from previous data”. 

In the only illustrative computations which he carries out, the value 
of Pq is obtained by dividing the area under a triangular-shaped 
plot of error rate versus time, by the total test time, 

Shooman then uses the total number of changes (the area \mder the plot 
of error rate versus time) which were observed during the entire test 
period for the value of E^, Coupling this with his assumption of a 
constant error rate, it would seem that the model is complete, hut has 
no predictive potential. Shooman uses it to compute the METF versus 
the debugging time, and illustrates the eventual unlimited magnitude 
for the MTTP, Since the MTTF is the reciprocal of the factor y in Eq^, 5, 
and since as t Increases, increases, the factor y decreases, and 

the reciprocal (the MTTF) increases (without limit). This is not, by 
any means, stirprising, since the cc(t) tends to approach Ej/I^, so 
the denominator tends to zero. 
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In the second presentation jointly •written by J, C, Dickson, J* L* Hesse, 

A, C, Kients and M. Shooman, a much improved, discussion is made [Reference 
In 19T3 and 1975 Shooman essentially discussed the same model in the same 
form given in the original paper [Reference 5 and 6]. He does however dis- 
tinguish between a microscopic approach, in which individual errors are 
studied, and the macroscopic approach (in which class his model falls) 
which treats bugs which are "lumped and treated equally". 

In the analysis of his macroscopic model he determines the unknowns K* 

(the product of K and rp in his original formulation) and by 
solving the equations 



K' 


% 


K' 


% - 


(6) 

(7) 


where and are two debugging times with and "^2^ * 

Xg, and Xgp are the ntmiber of software failures "found during total 
activation times” and H 2 , respectively. 


Each of these two expressions ecjuate a "steady-state" MTTF, on the left 
side, obtained by observing the error rate over a post-debugging interval 
(of duration t) and the factor 1 /yC of Equation 5)j which is the MTTF at 
the end of the corresponding debugging time. This is referred to as 
the method of moments. 


Prom the solution which is obtained: 
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it is clesr that the and Xgg represent the niunher of failures during 
tests made after and x 2 units of dehugging time and the Hi eind H2 refer 
to the ’‘locally" cumulative numher of hours after the debugging times 
XI and Tg and not the total activation times. This is an important clari- 
fication and changes the focus of question of independence of measurements. 

If these quantities were cumulative from the beginning (instead of 
locally) then the Xg*s are highly correlated, but If they are only locally 
cimiulative, then it can be seen that the independence question is now at 
the microscopic level; the new question is whether the individual measiire- 
ments which make up any particular Xg are independent. 

This couples with another question and there is a single answer to the two. 

The other question is: do the individual error-separation-times all estimate 

the MTTF in an unbiased way? The answer to both is the same: if there are 

a large number of incipient errors in the package, then the process indeed, 
resembles the "infinite number of failure-makers" of hardware reliability j 
and, for a short period after X2_ (or xg), the individual measures of time- 
between-errors are independent (the coin-flipping analogy) and are unbiased 
estimates of MTTP ( t) . Clearly, since the basic assumption is that the 
MTTF is (inversely) proportional to the number of residual errors, the number 
allowed for each Xg must be small, for only the first error found is 
strictly an unbiased estimate of the MTTP. 

The Xsp and Xg2 sre "the number of software failures" and as defined by 
Shooman are the number of unsuccessful ixins (not errors) caused by a 
software "failure", and Agg are defined by formula above and clearly 

represent the rate of failure, and not the rate of error-occtirrence. Thus, 
at best, Shooman is equating two different kinds of failure rate: on the 

left his (approximate) rate is the number of failures per unit of operating 
time, while on the right side the rate is the formula-derived ntmiber 
of errors per unit of operating time. 

It can be argued that is not the number of errors in the program, but 
some kind of measure of the number of incipient failures, CoTmtering 
this, however. Is the fact that in Shooman’ s formulation Eg? is divided by 
the total number of instructions It» implying that they were thought of 
strictly in terms of coding errors, (In the applications which were made by 
Jelinski and Moranda, it was necessary, because of a lack of fine-grained data, 
to analyze on the basis of "trouble days" instead of on error count, but 
there was not any consideration of program size in that analysis), [Reference 
1 ], 

As an important comment in this respect, it is clear from Equation 7 that 
the number of instructions is only a nuisance since it is taken out 
by the "normalization" of Sg (xj^) and (T2), which is required to 
produce numerical values. 
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B.2.5 Sohiefc-Wolverton Model (Description and Critique) 

George J, Schick and R. W, Wolverton, in Septemher 19T2j at Hamburg, Germany, 
presented a paper in which the de-eutrophication model was described along 
with a new model which uses the same notation but which describes an 
entirely different failure rate* 

The rationale of the de-eutrophication model, to quote from the original 
paper, is; ’’the failure rate at any time is assumed to be proportional 
to the current error content of the tested programj the initial error 
content is then denoted by N and the proportionality constant is denoted 
by 0, the failure rate drops to (K-l)0 after the first error is detected 
and 80 forth." 

Schick and Wolverton make the comment in their paper that; "there does 
seem to he an inconsistency by admitting a decreasing failure rate yet 
at the same time assuming {rather than deriving) an exponential model". 
Apparently, they interpret the failure rate of the Jelinski-Moranda model 
as applying to a single error rather than to the sequence of a number 
of errors. This interpretation cannot be supported by either the analysis 
of Jelinski-Moranda in their paper or, by Schick-Wolverton ’ s own analysis 
of their own model as described in their paper. 

To make the point more clearly, an examination of the defining equation 
is useful. In the model, the detection rate is given by 

Z(t.) = 0 [N-(i-l)]t^ 

where tj, is "the cumulative time to the occurrence of the ith error". Under 
that interpretation the variables employed in the likelihood function are 
not independent. On the other hand if tj_ represents the time past the 
occurrence of the (i-l)st error, then the represents the same measurement 
as the Xi of the de-eutrophication model; this is the interpretation that 
agrees with their analysis. 

In a purely formal way the likelihood equations for solution are; 

2n = E [K-(i-l)]t.^ (8) 

0 i=l ^ 

and 

n Up 

2 1 =0 2 t/ (9) 

i=l N-C'i-1)' 2 i-1 

where the symbols are the same as those used in Equation 2 (with X^=ti), 
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B,3 DATA AHD ADJOSTMENTS 

Data which is relevant to software errors was obtained recently from 
W* L. Wagoner [Reference 8]. This data, although not in a form which is 
ideal for analysis hy the three MDAC models since it consists of grouped 
error counts, was analyzed to obtain estimates of the error content. The 
unique feat\ire of the data is that the reference unit ms CPU time (in 
seconds ) , 

By adjusting the data in the ways subsequently described it was possible to 
obtain estimates for all MDAC-models. The results produced estimates 
which were consistent among the models and accurately predicted the error 
count which was eventually achieved on the basis of a very short interval 
of data. 

Because of the grouping of the data it is very easy to obtain estimates from 
the Shooman formulation in either of two interpretations. 

In order to complete the comparison, the Schick-Wolverton model is also 
employed in a formal way. 

The data consists of a record of the errors which occurred during the 
debugging of a data-r eduction program (called the Fil-D Program) consisting 
of "approximately 3-H thousand" Fortran statements. The data is reproduced 
in part in Table I, The important feature of this is that CPU time is 
available as a unit. 

The three MDAC models describe f.ailur e-rates (or Poisson parameters) which 
decrease with time. The first t^lear cut evidence of a decreasing failure 
(found hy dividing the errors detected by the CPU time on a daily basis) 
starts on 1/19 after 5.2H units of CPU time has elapsed. The ratio on 
l/l6 is 1,5^ while on l/l8 it is 10, 06 errors per unit of CPU time, while 
on the next three data-days the ratios are 2,oU, 1,31, and 1,10, 

This corresponds qualitatively with other experience which has been gained 
on other data. There is usually a startup effect which is evident. There 
are fairly clear reasons why this should he so when calendar time is the 
unit: early in testing there may not he a sufficient number of "working 

pai*ts" of the package to obtain significant error counts; as time goes 
on these parts produce in total an increasing error count; finally the 
assumptions of the models may be met. 

Although the same reasons do not necessarily apply to data based on CPU 
time, some of the genersil effects seem to he indicated by the data. 

It should he said, however, that experience also shows that the two de- 
eutrophication models can be applied to any initial segment of data. The 
estimates for the error content will be initially very high (infinite for a 
constant error rate) hut will settle down to give good estimates even though 
the first part of the data is not being well-modeled. 
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Table I 

DATA ON Fll-D PROGRAM 


Date 

Errors 

Detected 

Cum 

Error 

' CPU 
Time 

Cum. 

CPU 

Time 

1/12 

8 

8 

0.5 

.5 

1/15 

7 

15 

o»6 

1.1 

l/l6 

1 

l6 

0.65 

1.75 

1/17 

8 

2h 

1.90 

3.65 

1/18 

16 

ho 

1.59 

5.24 

1/19 

18 

58 

8.83 

14.07 

,1/22 

13 

71 

9 . 9 ^ 

24.01 

1/23 

8 

79 

7.25 

31.26 

1/^ 

9 

88 

8.34 

39.60 

1/25 

2 

90 

3.86 

43.46 

1/26 

6 

96 

13.11 

56.57 

1/27 

3 

99 

34.15 

90.72 

1/29 

3 

102 

82.7 

173.4 

1/30 

2 

lOlt 

1.10 

174.5 

1/31 

3 

107 

51.59 

226.11 
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On the other hand, in order to apply the Geometric -Poisson Model, anich 
greater care has to he taken, since there is much less data which can he em- 
ployed for the estimates of the two unknowns. 

The above reasons are all good reasons for choosing the zero time of the 
analysis to he the cumulative CPU time at the end of l/l8. But the way 
in which that time was initially chosen is entirely different. It was 
chosen initially hy applying a tmiversal ’'Programmers Poisson Parameter” 
of 1 error per 50 lines of instruction. In order to eliminate the startup 
effects on a program with 1^000 instructions and an estimated 80 errors, the 
zero time chosen corresponded to the half-way error 

While the particular value chosen does not seem to cause any concern, the 

use of the factor of 1/50 to obtain the a priori error content has caused 

controversy. As originally stated the rule-of-thumh is that there are 

(on average) two errors per 100 instructions. This factor has been observed 

hy F, Akiyama on nine fairly large programs. [Reference 9] It also has 

been noted hy B, W, Boehm of TRW in a presentation at the 197^ AFIPS 

Conferencet data from T, A-, Thayer, et.al,, taken from tests on five 

large scEile programs showed a remarkably consistent rate (22xl0"3) .[Reference lO] 

Boelm in a personal communication, stresses the important fact that the 
constant he reports is the ratio of errors (program hugs) to the number 
of source instructions (vis a vis machine or object instructions). But, 
fortuitously or otherwise, this is exactly the way the figure was employed 
above in estimating the half-error point (4000 Fortran instructions times 

1/50), 

Excepting the adjustment for the zero, the data is used as it stands for the 
two de-eutrophication models. In order to apply the data to the Geometric- 
Poisson Model, it is necessary to further adjust it. Time intervals of 
equal size and the number of errors per interval are required for this 
model. The choice for the length of the intervals is arbitrary (for 
illustrative purposes); however, five of the six daily CPU times in the 
time span 1/19 through I/26 are about 10 seconds in length, so that It 
is a convenient interval size for comparisons among the models. 

In order to apply the data to this interval size, interpolations of cumula- 
tive error versus cumulative time are required. As with the previous 
analyses, and for the reasons given there, the zero time for data corres- 
ponds to a cumulative CPU time of 5«2it, The data after interpolation and 
adjustment to the "new” zero, is shown in the first two coltunns of Table II* 

B,k ESTIMATION OF PARAMETERS 

B,U,1 Ue-Eutrophication Analyei£ > 

In the application of the de-eutrophication process the times between 
successive errors form the primary data. In the present case the data 
are not recorded in that way. As an expediency the times within any given 
interval are put equal to one another and have a value equal to the 
quotient of the CPU time used on a given date, and the number of errors 
found. 
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Adjusted 


The analysis is performed in this "vra-y on the errors recorded during tvo 
days (1/19 and l/22). Thus, the required data are; 

= X2=...=X3_g=.U906 


and 


X 19 = XgQ =,,.=X2^=.T646 

This data produces the numerically substituted version of Equation 1 
[Reference l], 

,1 = 31 

i=l N-(i-l) N-i'67fot"6~ 


which produces an estimated residual error of 63,4 which, together with 
the 4o "startup" errors, produces an estimate of 103,4 for the total error 
count. 

The estimate for ^ is obta^lned by substitution into Equation 2, of 
Reference 2 and has the value 0.035, 

If the same analysis is oaployed on the data for the three consecutive working 
days 1/19, 1/22, l/23, there are 8 additional errors and the time between 
them is taken to be equal. The additional data is; 

X32=X33=,..X39=,9065 

The numerical equation to be solved is: 


¥ 

i=l 


1 

N“Ci"l ) 


39 

N- 21 , 66513 


which produces an approximate solution of N-68,8, corresponding to an 
estimated total error content of IO8.8, 

The estimate of si in the extended data case, can be computed to yield the 
value, .032, 

The above estimates which are based on just two (or three) day^s data 
yield estimates which "turn out" to be quite good. The program after 
running seven additional days had uncovered a total of 107 errors. 

While it is very unlikely that all errors have been found, the time 
spacing of the latest errors recorded is very long, and the program has 
a "practical" error content of somewhere between 110 and 115, But it 
should be pointed out that the data employed in the prediction is only 
1/10 (or 1/8) of the total observation time, making the result even more 
noteworthy. 


B-13 


Geometric De^Eatroptiication Analysis 


The same data used in the preceding analysis is employed with Equations 8 
a n d 9 to produce estimates of k and D [Reference 2l, 

Using the data for the first two days (l/l9 and 1/22) produces estimates ; 
k = .9735 and D - 2.993. 

It is clear that this model cannot he used to estimate the total number 
of errors j however, it is possible to determine the level of ’’purity” after 
n observed errors by evaluating k^i. 

The estimated degree of "purification" after n errors have been detected, 
is given, generally, by the ratio Rp-Rf , where Ro and Rf are the initial 
and final rates. In this ' ^o particular case, it is l-k^^. 

For n=31 the degree of purification is 56, h%. The corresponding degree 
for the de-eutrophication model is given, in general, by the ratio n/R, 
and for the same data employed earlier this is 31/60,3 or about 51. (under 
a different interpretation, where the initial errors are included in 
both numerator and denominator, the ratio 71/100. 3 == » 708 , could be used; 
however, the former figure is clearly the proper one for comparison of the 
estimates of the two models as they are applied here) . 

B.4,3 Geometric Poisson Model 

As a final analysis of the same data, the Geometric-Foisson Model is 
employed. The data required has been described in the preceding section 
and consists of the entries in column 2 of Table II, Using these as 
the ni, and substituting into Equations 11 and 12 of Reference 2 
produces the polynomial equations 

2 . 4U33k'^-3 . 1320k^tl , 6887k-k=0 , 

which has a root k = ,6756, 

The value of the Poisson parameter is found to be =20,348, 

= 20.348, 

Use of these two parameters, produces the third column (labeled "Fitted") 
of Table II, 

By extrapolation (summing the infinite geometric series) the projected 
total error count is 62,73, (or with the 4o, which were "banked", a total 
of 102.73), This appears at first to be only a fair estimate, when it is 
compared with an observed total, of (at least) 67 , But it is important to 
note once again that the last time-point used in the analysis corresponds 
to a modified CPU time of 60 seconds, while the 63rd (or 103rd) error 
occurs after about I 68 seconds of (modified) CPU time. With this scale 
of reference, the estimate is remarkably accurate. The model data can be 
used to generate forecasts for each time period; for the 60-70, 70-80, 
80 - 90 , intervals, they are 1 , 93 , 1 , 31 , and , 88 , respectivelyj while the 
observed counts obtained by interpolation of the actual data are ,88, ,88, 
and ,64, respectively. 
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Shooman Model 


The Shoomn Model does not require a decreasing failure rate and the choice 
of the time Tj|_ is arbitrary. However, in order to achieve some compatibility 
with the two analyses made with the de-eutrophication models, the time 
Tn is chosen to correspond to the fiducial time, 5.2^ CPU-seconds, Choice 
or the time Xp is somewhat open, but to test the model thoroughly, several 
choices for are made, each of which will give an estimate of the 
error content , 


Colimn 3 of Table I lists directly e (x). For the quantities Xgj. and 
we employ the narrower, and more proper, interpretation that they relate 


only to a short time 

segment subsequent to 

with tt , are the quantities xsi = l8, and : 

2 and 4 respectively. 
Case X; 

Tl = 5.24, Tj 

= l4,07 (=3 times x^) 


e^(Tg)=5S 

= 18 

X,, = 13 

1 

®2 

H = 8,83 

Hg = 9.94 . 

Thus 

2,039, 

A^ « 1,307 
®2 

and substituting 

H, « 25,663-58 
^ 1-.64i5 

“ 32.337 = 90.22 

,3564‘" 

This compares to lOT 

errors found. 

Case II; 

*^1 s 5,24 

“^2 =: 24,01 

e(x^)=40 

eCxg} = 71 

x_ = l8 

X =8 

®1 

"2 

= 8,83 

Hg = 7,25 

A = 2,039 
®1 

A^ - 1,103 
®2 
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JL = (.541) 40 - 71 



.541 - 1 


l E 

= 107.54 



This is almost exactly the mmher 

found when the same data was used in the 

UJ 

de-eutrophication model, and agrees with the ohserved number of errors 


quite well. 


.'f 

Cii 

Case III; 


fTffM 

= 5.24 

= 31.26 


= 4o 

= T9 

rw 

X - 18 
®1 

X =9 

®2 

Cw 

ffr^ 

= 8.83 

= 8.34 


Thus 


**•T^ 

•5 /• 

= 2.039 

®1 

A = 1.079 

^2 

u if 

= 122.8 


r- (• 



-T 

Case IV; 


3- 

:ij; 

Oi/ 

= 5.24 

= 39.60 

"F 

e^(r^) = 4o 

E^CTg) = 88 

;■ ir 

■st; 

t?iJ 

X = 18 

X =2 

i; h 

w » 
1 

®2 

:■]. 

= 8.83 

= 3.86 


Thus = 2,039 

A„ = .518 

Sg 

rrn 

. 1 . 

\V'^ 

LaJ 

- 104.358 


Kli/ 

f'V>l 

. ■ j' 

m 

■• ')i 


- 

-n, 

■ > ‘‘ i 
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Sehiek-Wolverton Model 


The technique requires solution of the equation 

n T _ ESt.^ 

2 J, “ 1 

i=l W=i+1 E(N-iti)ti2 

where all variahles and parameters are previously defined ^th tj^ 
replacing 

Using the same data as employed above for the tiro de-eutrophication models 
the following are obtained: 

^^2 2 2 

£ ti = i 8(.4906)^ -f 13(.T6U6)^ - 

i=l 

= 11.932 

r(N-i+l)t.^ = N g t*^ - F 
i=l ^ i=l 

IT 

= 11.932N - C i) 

= 11,932N - 219. 2T 
The estimated error content is the solution to 

1 = (11.932) (31) 

i=l iT:!?! 11.932N-219.27 

By trial and error the solution is 

N = U.5 

Again since 1^0 errors are "banked”, this corresponds to an estimate of 8l,5 
for the total error content. This does not appear to fit the error process 
very well. 

B.U.6 Siimmary of Error Estimates 

The results of all models are extremely encouraging when viewed in the large. 
The "gestalt” which seems most important is that "nature” does indeed relate 
residual errors to the MTTF in sn inverse way: all models are based on 

this assumption in one way or another and they all produce reasonable results 


(i-l)t^^ 

2 

(.4906)^ - (|^^g i) (.7646) 
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B.5 QUANTITATIVE COMPARISONS OF MTTF ESTIMATES 

As noted before tbe estimates of MTTF at the end of test time provides a 
”close-in" estimate. All models, except the Geometric-Poisson, which is 
based on several time intervals, can be compared although they formulate 
estimates of MTTF in different ways. The comparison can be made against 
the realised MTTF for the time (or times) concerned. 

B,5,l De-Eutrophication MTTF Estimate 

Using the estimates for N = 63,^ and ^ = ,033, the natural estimate for 
MTTF at the end of the two-day sample is the reciprocal of the hazard 
rate, (N~n)^, 

For the two day sample (using cum CPU time of Table 1 ) 

MTTF (2i}, 01) = .88U (Actual Value .906). 

For the three day sample ending at CPU time 31.26, there results 
MTTF (31.26) = 1.0i^9 (Actual Value .927) 

B.5. 2 Geometric De-Eutrophication MTTF Estimates 

For this model the MTTF is the reciprocal of Dk^ where for the two-day 
sample D = 2.998, k = .9735 and 

MTTF (2U.01) = .76'^ 

For the three day sample, D = 2.520, k = ,97^^ and OTTF (31,26) - I.09I 
B.5.3 Shopman Model MTTF Estimate 

For the two-day sample. With using - 90.22, ^ ~ .0U06, 

MTTF (2U.01) =1.283 (Actual Value .906) 

and for the three day sample 

= 107. 5it, e(T2) = 79, c = .0302 
MTTF (31,26;^ 1.161 (Actual .927) 

B.5.4 Summary of 1-CTTF Estimates 

The following table provides comparison for the MTTF estimates. 


Table 2 

MTTF ESTIMA.TE COMPARISON 


CPU 

Time 

De-Eut, 

Geom. 
De-But . 

Shooman 

Actual 

21:. 01 

.884 

.76? 

1.283 

.906 


(2,4^) 

(15.3^) 

(4l.6j5) 


31,25 

1,o49 

1,091 

l,l6l 

.92? 


(13.5/^) 

(IT.T^) 

( 28 . 1 ^) 



The main entries show the estimate for the corresponding model, and in 
the last column, the "actual” value obtained by dividing the number of 
errors by CPU time for the day just beyond the test truncation time. 

In parenthesis are relative errors in percent, 

B.6 SENSITIVITY OP ESTIMATES 

A proper comparison of the models with respect to their robustness of their 
estimates in the presence of changes in assumptions can best be done by 
simulation. However, a very simple indication of the behaviour can be 
found by employing properties of maximum likelihood estimates. In particular 
the variance of the estimates can be approximated by means of an asymptotic 
formula developed by R, A, Fischer, This formula and the separate analysis 
for each model are given in the following sections, 

B,6.1 De-Eutrophication Process 

The analysis for this model is based on the following likelihood function: 

J ?i[N-{i-ll exp {-«! [N-(i-l)]Xi> (10) 

i=l 

where i and N are the parameters previously defined and is the time 
separation between the (i-l)st and ith error. 

By partial differentiation of the logarithm of the likelihood function 
the Maximum Likelihood Equations (MLE's) can be fomied. They are: 


81ogL/aH 


( 11 ) 


and 


91ogL/a?i 


n 

E 1 

i-1 N-(i-l) 


n 

E ?5X = 0 


n 

n - 2 [N-(i-l)]X. = 0 

i=l ^ 


( 12 ) 


The variability of the estiraatea becomes of interest when an attempt is made 
to compare different models. Obviously the comparisons of models must be 
done on the basis of additional factors and by repeated applications on 
similar data, Nonetheless, other (unspecified) things being equal, the 
model which provides the smaller variation (standard deviation) is preferred 
to others. 


Unfortunately, because of the implicit nature o'* the solution to the MLE’s, 
the probability distribution(s) (joint, or marginal) for N and ?! cannot be 
obtained, but this difficulty can to a degree be circumvented. 


The general properties of maximum likelihood estimates can be used in a 
purely formal way to derive some measure of the variability in the estimates. 
This point must be emphasized since it is manifest that the use of asymptotic 
formulas (involving large sample sizes) on samples which are fundamentally 
limited to be finite (there can be no larger samples than there are errors) 
can result only in caution-laden approximations. Nonetheless, the experiences 
which have been gained using the models seem to indicate that these approxi- 
mations for the variances are generally much too high. 


The basis for the development of the large sample estimates is a theorem 
due to R, A, Fisher which states that under certain "general conditions", 
which have to do with the boundedness of the first three derivatives of 
the likelihood, the variance and covariances of the estimates are given by 
the inverse of a matrix formed from the mathematical expectation of second 
partial derivatives. Explicitly the matrix Aij (which is to be inverted) 
in tt= estimation of several parameters ( 2 s*.,s terms 



(13) 


where L is the likelihood function and 0^ and 0, are two of the 
parameters. From Equations (U) and (l2) above ^ 

9^ = - S 1 (lU) 

3^2 i-1 (N-'i+iP 
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(15) 


3^ = 

2 

= n = - 

£ 

3N3?4 

3^W 

i=: 

dh, 

= - n 





And since 



E(X. ) 

= 1 


1 

(N-i+l)?! 


the matrix 

elements become: 


n 


^1 ” 

i~l X¥-iViT^ 


n 


^12 " 

^21 = ?, 
1=1 

(n- 

= 

n 


22 




X. 


where for evaluation in practical situations, the values of N and ^ 
(the estimates based on the data) are used. 

The 2x2 variance /covariance matrix can be simply computed. 

The determinant (denoted Det of the A-matrix is 

n 


where we have used the fact that "on the average" s 


n ~ T‘ 

5T- 


n 

2 1 

(i\’-i+l)(ir 


=: T, the total observation time. 


Hence 


Var (N) a n 
^2’ 


(l6) 


(17) 

(18) 
(19) 


( 20 ) 


( 21 ) 


Deti 
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X 


I 


Var (p5) 



^ 

N-i+1 


2 



(22) 


4b A 

Covar (l/,^) = - T 

Det^ 


(23) 


Sinoe for a fixed sample size n, the solutions for K and i> hy means of 
Equations (11 ) and (12) depend only on the ratio R - £(i"l}Xi , it is 
possible to tabulate solutions as veil as the 
variance and covariance. This is illustrated by Table 
III which shows the values for a sample size n=26. 

In order to tabulate the parameters for an arbitrary process it is necessary 
that the scale for time be normalized. Since the total observation time, 

T, is assumed recorded by the data collection process* it is a natural scale 
factor to use. It must be pointed out however, that this time is a random 
variable; although it is treated as if it were a constant, this is a purely 
pragmatic interpretation, A reasonable interpretation which can be made 
is that the results which are recorded are conditional on the observed time. 


Given the ratio R, the MLEs become 


n 

?, 1 n 

N-(i-l) M 

and 


flJT = n 
R-R 


(2l+) 


(25) 


Equation (2li) can be solved essentially by trial and error. Once N is 
established the quantity (iT can be obtained from Equation (25). The 
quality ^ is entered in column 3 of the table. 

The variance of R can be obtained in the following way: 
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Column 1 is the ratio S{i-l)Xi / 

CoXumn 2 Is the estimate for the total error content 


Column 3 is the normed- estimate for step size: in order to determine the 

actual estimate of the step size, the entry in this columr should be divided 
by the total observation time T, 

Column 4 is the approximate standard deviation of the estimate of the total 
error content* 


Column 5 is the normed standard deviation of the estimate of tha step size: 
in order to obtain the actual standard deviation the entry in this column 
should be divided by the total time T. 


Column 6 is the normed covariance between N and In 
actual estimated covariance the entry should be divided 


order to obtain the 
by T. 


Column 7 la the normed MTTF and in order to obtain the 
entry should be multiplied by T. 


actued value the 














1 


1 




5 I,-..- 





Ljii 


Ljy 


"b-jE Equation (21) 
Var(H) = 


XX 


"but using the substitution 
n 

Sp £ 1 

i=l Xi-i+i)2" 


( 26 ) 


(2Y) 


the determinant of Equation (20) can be expressed as 
Det^ = ^2 - 

or 

ii^et^ ^ nS^- {?iT)^., 

Hence the denominator of Equation (26) can be evaluated using the estimates 
and H. 

Hence 


Van (^ ) = n 

nSg - '(?5 t)2 


(28) 


The standard deviation ig the more useful measure and is obtained by tflxking 
the square root of Var (k). This is entered in column of the table. 


The variance of ?T is obtained in much the same way: Det^ is evaluated, 

as before, and with S 2 as defined, 

2 

S^T 

Det: 


Var(^T) = T' 


(3a)' 


nS^r TOTT- 



: V 


eU 



liU 



tail 






u 




rt-tv 

I 


(29) 




i 


•i ‘ 

;l: 

do 


1 ; 
i ■ 


\ 

i 


I’’ 

I ' 


\ 

1 ^^ 


The standard deviation of this quantity is computed and listed in column 5 
of the table. As noted a footnote the standard deviation of ^ is obtained 
by dividing the column entry by T, 

The covariance between K and ?5 t ■'s obtained by 

Covar (H,?!T) - - T^ « - (^T)^ (30) 

Det^ nSg - 

Again for the covariance between N and ?5, the entry in column 6 should be 
divided by T, 

The MTTP of the "next” error is given by [(N-n)pJ3” and is estimated by 
employing ^ and The value entered iu the last column of the table 
(for n-26) is [ (i?-26)^T]-l and so must be multiplied by the user-found T* 

While it is possible to formally express the vaiiance of the estimate 
of the MTTF in terms of the variances and the covariance of the two 
estimates this is a step which will not be tedcen as the cascade of 
approximations is already too long. 

In the next section \7here the model has estimates which have legitimate 
asymptotic properties, the variance of the MTTF- e at imat e is given. 


B,6.2 Geometric De-Eutrophication Process 

The analysis parallels that made for the De-Eutrophication Process, 

It is important to note, however, that this process has an unlimited number 
of errors. The likelihood function for the sample Xj,X2 , « , . ia 

L = g Dk^“^ exp {-Dk^'^X.} (31) 

i-1 ^ 

and its logarithm is 

logL = nlogD + logk-^ -D ^ 


The MLS’s are obtained by differentiation and reduce to the two equations 


n 

D 


n 

? , k 
1=1 


^-tc. 


(33) 


and 


1 

k 


n 

L 

i=l 


(i-l) = D 2 (i-l)k^"^^, 

i=l 


i3k) 


B-25 




1 





D can be eliminated from both equations to leavv= a single equation. 


(' 

u i 


n 

Z • 1 ^ 
1=1 ^ ^ 






ntl 

2 


(35) 


Then using the solution, denoted k, the estimate for D is 
D = n , 

The Tariance and covariances for this process are found by the procedure 
described above. As noted, above, however, this process has an infinite number 
of errors, and so the sample size can become large, and the asymptotic 
formulas can be applied without apology. 


Directly by differentiation 


a^iogL 

« - n 


d>Ho^ 

« a^iogh = - 

n 

2 

3D3k 

3k3D 

i=l 

3^1o^?L 

n 

= - 1 2 

(i-D- 


^2 i»l 



(i»l)(i-2}k^“2x^ 


(36) 


(3T) 


(38) 


Since E(X.) = 1 , the associated A-matrix elements are 


Dki-x 


A - n 

D^- 


^^o = = 1_ ? . (i-1) = 1_ n(n-l) 

Dk Dk 2 


(39) 

(Uo) 


'22 ' 4 ill (1-1) * ill (i-l)(l-2) 


= 1 2 (1-1)*=^ = 1 n(n-l)(2n-l) 


i=l 


6k" 


(Ul) 
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Using Det 2 to represent the determinant of the A-matrix, we obtain after 
simple reduction: 

Det- = 1 . n^(n^"l) (^2) 

2 2 

D k 12 

Thus the variances and covariances are 


•t . 

; 

i » 

Var D = 2(2n«l) 

(43) 

n(n+l) 


i 

1 

Var k = k^ 12 

(44) 

3 . 



1 ■ 

( 

i 

A 

Covar (D,k) = -Dk 6 

n(n+l) 

(45) 


I 

i 

1 . 


I ■ 



In the limit these variances tend to zero. On the other hand, it will be 
noted that the correlation coefficient between the estimates is quite 
high (in absolute value): 


P = - ^3 Xn"-Tj’" 

2 '(2n-l') 

which is in excess of 0,85. 


(H6) 


The estimate for the MTTF which has the character of the maximum likelihood 
estimates is given by 


i 

( 

i 


^2 = 


Dk^ 


n 


where the subscript 2 denotes the estimate for the GDEM, The asymptotic 
approximations can be employed in another reasonable approximation in 
order to derive a measure of the variation in the estimate of the MTTF, 

By differentiation taking the total differential and expectations it 
is seen that (Uy) 

Var Mp = 1 Var(D) + 2n Covar (D,k) + n^ Var(k) 


where, again the estimates would be used as proxies for the (unknown) 
parameters. 
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B,6,3 Geometric-PolsBon. Model 


From Equatiioii 10 of Reference 2, the likelihood function is 

m 


L(n, ,n„,..,,n„) = H Uk^“^)“i exp(-Xk*""-^), 1 

^ “ i=l nA 


i-1, 


and 


^ _ 

logL = S n. (logX+(i-l)logk) - E Xk"^ - E logti. ! 

i=l ^ i=l !■=! 
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i-1 
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Hence 
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^logEi 
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&^lopiL = 
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(i-l)n^ 

m . o 

- XE (i-l)(i-2)k 

(52) 

V 

“2 i=l 
k 

i=l 



I ( 


The variables assumed by this model to he Poisson distributed, 

and so 


E(n. ) = Ak 

X 


i-1 


(53) 


1 j 




Accordingly 

^ i-1 

A = 1 E k 
\ i=l 


^21 “ ^12 = ? , 
1=1 


Ag2 = XS(i-l)V'*^ 


(54) 

(55) 

(56) 




The variances and covariances are therefore 
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A . E (1-1)^^“^ 
Det^ i=l 


(5T) 
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B,6,k Shooman Model 


M, L, Skooman in Reference 5» gives the folloving expressions for the maximum 
likelihood equations for the tvo parameters or and C (originally K) : 


and 


C = 


°1 "^^2 
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H, + 
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+ ^2 



[Eq, 21 of Ref, 53 

(80) 
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[Eq. 22 of Ref. 5 I 

(61) 


where n^^ and ng are "the numher of runs used in testing for times and Hg 
respectively", (This is incorrect j they should he the numher of unsuccessful 
mns as will he shown herein), 

ShooDmn refers to another of his papers for the justification and development, 
unfortunately thei’e is no discussion in the referenced paper with respect 
to maximum likelihood estimation [Reference 53 • la order to expedite the 
discussion and further clarify the Shooman Model, the following analysis 
is provided. 

Fundamental to the analysis is the assumption that the software error 
genera ion is governed hy a Poisson point process whose intensity (failure 
rate) is proportional to the current numher of errors. For short periods 
of time, (Hqj^ and Egg in the present instance) the intensity factor can he 
assumed to he proportional to the numher of remnant errors, which in 
Shooman' s notation is 

= C ( - e.p (t^) 
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for the first period, and 

^2 = C (Sj - Eg (Tg)) 


The numher of xmsuccessful runs occurring in a time Hg is then given hy the 
Poisson diatrihution. Explicitly, the number of errors (or unsuccessful 
runs or vhatever rare event is being coimted), ni, observed during Hg^, 
is given by the Poisson law 

P(N=ni) « U-Hg-)^le ^ (62] 

1 


Thus for 
ful runs 


two runs of duration Hg and Hg with observed numbers of unsuccess- 
and ng, the likelihcAd function is 


n, A,H„ 

L == (Aj_Hg ) ^ e ^ ®1 


(A„Hq ) e 2 


The maximum likelihood equations obtained by differentiating the log- 
likelihood are 


aiopji = n^ + np - %^1 

- - 


= 0 


31o^ ~ ^2 

^ ^T^l ^T^2 ' 


1 H G-1 H C = C 


where "used for convenience, and Hg is now 

T~ ^ 

replaced by 


B-30 


Solving for C in eacli eguation produces! 


C = 


and 


V2 


^2 ^2 


( 66 ) 


c = 


V^2 




Rh 


(67) 


and vhere R. and C are the estimtors of C and respectively. These are 
the equations Shooman reports. 

Following the same steps employed in the preceding analyses with respect 
to development of variances and covariances, the second partials are taken. 
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The expectation of n^ is 

X.H, = H.CR., so that the A matrix elements are; 

XI XX* 


^ 12=^21 = i 


^22 “ ^ 2 
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^ !a 

R, R« 


(71) 

(72) 

(73) 
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Hence, from general relations. 




Detj^ C 


1 + Hg . Rg] 


Var C = 1 

Deti 


£2 ^ ^ 

^ =1 82 


Oovar (6,%.) = - 1 . 1_ (1+B„) 




Detj^ = 1 2 (H^,R^ t Hg.Rg) . \\ + 

^ L \ 


~ 1 2 ^h*V 

T 


In Reference 5* Shooman gives the asymptotic relations 


Var C C 

n ■ ' •sfUMi 

and (in our notation) 


(Eq.e 23 o±’ Ref. 5) 


n 


" 1^1 


(•Eg., 24 of Ref. 5) 


These do not appear to he the same or simi3.ar to the corresponding expressions 
(T4) and CT5). if it is assumed that the second term in expression (76.) is 
much smaller than the first term, and can ce ignored, then by substitution 


^ r\j 

Var C = 


HR •i* H R 

"l”l “22 


If the expp'" -dd values for n^ and Hg we substituted into the Shooman expression 
these are seen to be the same. There is ho need to have n large, as Shooman 
implies in his formulation, but it is necessaay for the second term to be 
small. 


Correspondingly, using the same approximation for Hetj^ 


« /u 2 
Vai- = I„ 
T T 


^1 ^ 

^'1 ^2 


(T9)' 




If G is replaced lay its estimator given in Equation (66) above, this 
becomes 


A. p 

Var = I^^-C 




g+agR^ 


(80) 


This is somewhat similar to the Shooman as 5 )resaion but differs in a very 
interesting way: the terms involving products of n and R are "mixed", 

i*e., one factor is at and the other at 

There is an independent way of verifying expression (77) « From the knowledge 
of the Poisson-lavr, it is true that the variance of is equal to the 
parameter , so that equation (65) can be used to obtain 

Var C = 1 . x [Var(nj^) + Var (ng)] 

^ Vl ^2^2^ 

(there is independence between n^ and Ug) and 


Var C = C 

H^Rl+ HgRg 

which is equation (77). 

There is no explicit solution for so that a similar validation does 
not seem possible. 
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Appendix C 
PROGRAM OJESajING 


C.l IUTR0DUC2I0N 

Por the pmrpoges of this study, program testing vill he defined as the 
process of verifying that a selected self-rcontained unit of code (such as 
a suhroutina) complies "With, the requirements against vhich it uas 
designed, These requirements define the functions the code should perform, 
the structure of the code vithin the unit and the environment in vhich the 
code must operate. Program testing as Tja have defined it does not include 
the testing of relationships hetireen units of code. It is hased on complete 
hnouiedge of the interjial structure of the unit, as opposed to the hlach hox 
approach. 

It is well^hnown that comp3.ete testing is not feasible even in the few 
cases where it is possible* Therefore, the next best approach is to test 
as completely as possible vithin the constraints of economics and value 
received. 

Various tools and techniques have been devised to aid in program verification, 
"While none as yet can assure that the code performs correctly under all 
conditions to which it may be subjected, each adds to the probability that 
the code will perform its intended functions correctly at the time that 
it is called upon to do so, 

A number of tools have been built that support the testing function and 
research continues in the development of technologies that will result in 
new tools and in improv^ents of those already in existence. Some manual 
techniques have been advanced which offer promise as verification aids. 

The tools and techniques with vhich this tusk is concerned are those that 
actually interact with the code in some way as opposed to those that 
support testing in a peripheral fashion or those that support system 
testing from a functional standpoint. 

This report des^s with the application of these tools and techniques 
to the testing of programs , the research being performed to improve the 
technology, and an evaluation of the practicality of each type of tool 
or technique in tod^^ s' software . 

C.2 MODUMBIZAT 

The testing of 'programs using .curren'ily available tools and techniques 
. general3^>- requires; that ;^t^ total system be brchen down .into manageable 
•units, each of which can "be considered a • sepSTate test 6b j ect , Top-down 
design^ultimatfily ; resu^ in a set of routines . for each level .of; deqompositi^^ 


Each routine (unit) is a test oh^ect to which the tools and techniques 
can he applied* Intellectual manageahility of the unit is one criteria for 
establishing the size of the unit of code, particularly for manual techniques 
such as walfc throughs and program proofs. A second critex-ia is the siinount 
of code that can be handled by some tools such as test case generators. 

Other criteria may be considered such as requirements for single entry/single 
exit units of code. In most instances, limiting the size of the unit of 
code to lOOl statements or less provides a test object that is intellectually 
manageable and can be handled by the test tools which are to be applied. 

^ile modularization is a design function, its impoi*tance in program testing 
is so great that if not provided for in the design phase, it must be con- 
sidered in the testing phase. Decomposition of programs, while generally 
based on structure, must also take logical processes into consideration. 

C.3 MANUAL TECHNIQUES 
Walk-throughs 

The careful reading of a px’ogram by scaaeone other than the programmer 
with the objective of evaluating its correctness with respect to a 
given specification has proved to be effective, as documented by 
Weinbergl and Baker 2. 

The person reading the program can detect errors transparent to the 
programmer, because he is not psychologically biased by his identifi- 
cation with the program. The programmer who made the error will often 
consistently overlook it because he is reading the program from the 
same point of view as when he wrote it. 

The process of cooperative program checking is called egoless programming 
by Weinberg because it eliminates the egc identification the programmer 
has with the object of his creation and allows an impartial evaluation. 

The effectiveness of a walk-through is also a function of the ease with 
which the program can be understood. Since reading the program for 
correctness is in effect a mental proof of correctness, it is imperative 
that the unit of code be small enough to be understood. The mental 
execution of the progrem differs from program proving in the degree of 
formality, 

yalk-throughs can be performed at various stages of coding depending 
on the complexity of algorithm being Implemented, In very complex code, 
such as that which must meet extremely tight efficiency requirements, 
it may be advantageous to walk through the detailed design, the initial 
implementation, and the final refinement in order to assvre under stand- 
ability and reduce the chance of implicit or hidden errors. 
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Program proving 

Program pi*ov3.ng, both formal sad informal, is disciissed in Appendix E. 


When program proving was first seriously advanced as a candidate for 
automation, it vas thought that the difficulties inherent in this 
approach to program verification could be overcame given the time 
to consider them properly. As late as 1972 in a report by Information 
Research Associates ,3 it was stated that '*it seems possible that within 
the next two to five year period to bring this to a fairly respectable 
state of automatic analy^sis”. However, to date, the difficulties 
have not been found to be surmountable, and it appears that automatic 
program proving still faces some formidable problems demanding further 
research. 

As a manuEil technique, program proving can be considered from two points 
of view. The first is from the point of view of proving an algorithm 
correct and can be performed diiring the design stage. The second is 
from the point of view of verifying the design representation in code 
and Is performed during the coding stage. 

For units of code containing more than a few branches and/or loops, 
the manageability of the proof becomes virtually impossible. Therefore, 
while the theory behind progr^ proving is viable, its use is not 
feasible until viiys are devised to more fully automate the process 
in a workable fashion, 

C.4 AUTOMATED TECHNIQUES 

A variety of automated tools to support program testing have been built. In 
addition, a concerted effort is in progress to build tools with new capabil- 
ities or to add improved capabilities to the tools now in existence. A 
Gcmprehenslve evaluation of the software requires both static analysis to 
evaluate structural characteristics, and dynamic analysis to evaluate 
behavioral characteristics, While neither type verifies that the unit of 
code performs the functions specified for it, both provide useful information 
about the testability of the code. 

An interesting concept should be mentioned here, however. The kernel of 
functionel. requirements verification is the dynamic analysis of code if it 
can be determined that the unit meets the requirements defined by assertions 
added to the code. This capability is proposed as an enhancement to MDAC’s 
PET^ program, and is discussed in greater detail later in this report. 

An evaluation of the capabilities of various tools now in existence was 
made in the performance of this study and is contained in Appendix A, 

This task will consider the technology that supports these tools as well 
as others being developed in research projects. 
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Standard s Checkers 

'• ft i,^m 


The test object of standards checkers is the set of source statements in 
a self-contained unit of code such as a subroutine or a procedure,. They 
check the l -atement set for conformance to predefined standards such as 
adequate and consistent commentary, statement positiDning relative to the 
entire set f.e,g, placing declaratives in a specified order, placing format 
statements together before or after the ei'jecutable code, or placing interhal 
procedures before executable code Cin a procedure-oriented language) , ^d 
program length, 

5 

TRW’s Code Auditor program currently checks for 38 programaing constraints 
in FORTRM code, 

6 

Applied Data Research (ADR) has COBOL standards checking capability in their 
Metacobol system, 

T 

Computer Software Analysts, Inc, (CSA)‘ markets a product called Standards 
Auditor which is available for both FORTRAN and COBOL source programs* 

Q 

Bellas PFORT Verifier checks FORTRAN code for conformity to ANSI standards. 
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While it may be argued that theoe tools are not directly involved in pro^am 
testing, they impose an orderliness upon the program that is designed to 
eliminate errors caused by haphasard program construction. This orderliness 
also contributes directly toward the program's under standability and main- 
tainability. 


Automatic Test Case Generation 


iiii 


The overview testing of software has been called an art because the selection 
of a set of test cases, that will adequately test a program with carefully 
chosen input data to minimise the number of cases and maximise the value 
of their application, requires a great deal of insight and cleverness. 

The development of a methodology to automate this process removes it from 
the realm of art and the implicit errors and omissions that are inherent. 

Several systems are presently being developed to automatically generate 
test cases, L, Stucki, MDAC and W* Howden9sl0,ll,12 of the University of 
California at San Diego are developing a test case generator for MDAC under 
contract to the National Bureau oi Standards, R, Hoffman^35l^*15,l6 of TRW 
in Houston is developii^ the Automated Test Data Generator for the NASA/Johnson Space 
Center in Houston, E, MiHerlT,,fo.\*merly of General Research Corporation, 
designed a test d&ta generator for commercial use in their automated tool 
collection called HXVP, L, Clarke2.8 of the University of Colorado is 
developing an automatic test case generator supported in part by an KSF grant. 

All of the above support FORTRAN programs, and are designed to generate 
test data based on an examination of the syntax. Other t^ea of teat data 
generators are in use, and are basically driven by input parameter s supplied 
by the programmer, with no direct knowledge of the source code being required. 

This report will address the work by Hoffman, Howden, Clal'ke and Miller since 
that work seems most applicable to NASA needs. 
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The systems liated abpye are presently heing developed to automatically 
generate test data hased oh aa analysis of the code. This syntactic analysis 
provides cases that depend upon the control structure of the program. The code 
is analyzed and the control structure identified based on the presence of 
predicates* These predicates (logical decisioas) cause transfers to various 
parts of the progha>^5' forward and backwards ®hd create the existence of 
a number of ^ternate paths through the code. 

If the branches dictated by the predicates were all Independent of each 
other, an enormous hinaber of paths would be possible in a program. Looping, 
in particttiarji has the Neatest effect. 

However, in: reality, the number is much smaller, particularly in a 

noa-iteratiye system, because the branches tend to be dependent. That is, 
a branch taken on a true condition may eliminate not only the false branch 
but an entire section of a path including other branches that can only be 
reached if^ the false condition existed for the first branch. 

The goal of the test data generators developed to date is to exercise not 
orly a^ statements in a program also all branches (It is possible 
to exercise all statements without haying exercised all branches). 

The systems to be discussed in this report all begin by defining the control 
structure of the coda, then they eliminate paths which cannot be executed, 
leaving on3.y the paths which can he traversed using some input data to drive 
execution down those paths. 

Semantic analysis is not addressed. It is up to the programmer to determine 
if the cases generated to exercise the code do in fact demonstrate that the 
code iUnctions as it was intended. 

The usefulness of automatic teat case generators lies in their ability to 
show that code as written can he reached when driven by some data within the 
input domain. The extrapolation of this information to program correctness 
within acceptable bounds is left to the programmer. The test case data 
can be analyzed to determine relevance to the areas of interest, providing 
the base for the proper set of test cases required to adequately exercise 
the program for the purposes intended. 

General Approach • 

f - Q To Tl T2 

The McDonnell Douglas approach, originally taken by Stuck! and Howden * * * 

was to decompose a HORTBAH. progr,aia into a finite number of standard classes 

of program paths, thep/tb try to generate a set of test cases that causes 

one path from each clla^s to be tast^. 

The methodology was divided into five phases* - ■ 

1) Analysis of a program and construction of descriptions of the 
standard claasea of paths * 

2) Construction of descriptions oif the sets of input data that cause 
the different standard cinSses of paths to be followed. These 
are implicit descriptions. 


3) Transformation of the implicit descriptions into equivalent 
explicit descriptions* 

4) Construction of explicit descriptions of subsets of the input 
data sets for which the third phase was unable to construct 
explicit descriptions* 

5) Generation of input values that satisfy explicit descriptions 
by the application of inequality solution techniques* 

In testing a program it is necessary to choose a finite set of paths that 
could be tested from the potentially infinite number of paths possible 
through the program, A boundary-interior method was used for choosing the 
paths. This method groups the paths through the progrsm into a set of 
classes. One path in each class is tested, by which it is assume^d that if 
a test is successful, all other paths in that class are considered correct. 

The philosophy underlying the boimdary-interior method is based on the 
assumption that a "complete set of tests must test alternative paths through 
the top level of a program, alternative paths through loops and alternative 
boundary tests of loops", A boundary test of a loop is a test which causes 
the loop to be entered but not iterated. An interior test causes a loop 
to be entered and then iterated at least once* 

The boundary-interior method separates paths into separate classes if they 
differ other than in traversals of loops. If two paths Pi and P2 are 
the same except in traversals of loops they are placed in separate 
classes if 

(i) one is a boundary and the other an interior test of a loop 

(ii) they enter or leave a loop along different loop entrance 
or loop exit branches, 

(iii) they are boundary tests of a loop and follow different paths 
through the loop* 

(iv) they are interior tests of a loop and follow different paths 
through the loop on their first iteration on the loop. 

Class descriptions consist of branch predicates, assignment statements, 

I/O statements and FOR-loops (to represent an arbitrary number of traversals 
of a loop,) The complete set of class descriptions for a program can be 
represented in the form of a description tree , in which the left-most 
path describes the class of all paths which test the interior of the loop 
in the program, and the other paths are boundary tests. 

The description tree is formed as the program is read with alternative 
paths being consutrcted for each branch and each loop. 


Once the set of paths has been deacrihed, the methodology of phase tvo 
constructs implicit input data descriptions of the sets of data that cause 
classes of paths to he folloved, The predicates and predicate affecting 
statements are extracted from class descriptions. Output statements are 
deleted and all input statements are replaced hy assignment statements in 
•Which dummy variables represent the values in the input stream (and the 
program is assumed to read the next ■value in the stream) , Phase two 
also deletes all assignment statements that do not affect predicates. 

This is done by reading "backward from each predicate and constructing lists 
of those variables that affect the predicates, 

Plmse three attempts to transform implicit input data descriptions into 
e;!^iicit descriptions. The assignment statements are evaluated by sub- 
stituting -the current symbolic values into the sta-tJement, which gives the 
dependent variable a current symbolic value. These values are then sub- 
stitu'ted when •fche variables are encountered in predicates and relations, 

-toy of these statements that do not affect the predicates deleted. 

The FOR-loops are processed in basically the same •way. If a loop is closed 
and does not change any variable affecting predicates it can be deleted. 

Problems arise when array reference and FOR-loop indexes cm only be deter- 
mined at execution times when veOues of variables occurring inside a loop 
are computed outside of the loop; when values of variables occurring outside 
a loop are computed inside the loop; and when disjunctive and recurrence 
statements occur. In each of these cases the values of the assigned 
variables must be classified as indeterminate and cannot be deleted. 

The evaluation is an iterative process, since statements in a loop may not 
he evaluatable until later processing. Once evaluated, re-evaluation of 
prior statements may be possible. 

Phase four completes the transformation of implicit descriptions to explicit 
descriptions by choosing particular values of loop bounds Md particular 
terms in disjunctive statements. These values are a subset of a set of 
values that satis^ the description. 

At this point, feasibility of a description becomes an issue, A description 
is feasible if there are values in the input domain that satisfy the des- 
criptions, Infeasible descriptions describe the empty subset of the input 
domain. Assuming the partially explicit description is feasible, Phase four 
attempts to choose loop bounds and disjunctive terms that result in a 
feasible subset description# 

Phase five generates the test data. It divides the standard classes of paths 
into three sets; those for which it can generate test data, those for 
which it can determine infeasibility and those for which it can do neither. 

Phase five is an integrated collection of lne0iality solution technigues 
and is based oh a backtrack search. This method can be applied to "both 
linear (Kuhn 195^5) £^<3. non-linear (Howden 1972) systems. 

The backtrack search starts ‘With the last inequality and progresses ■up 
through the path, solving each inequality in. sequence. If at any point, 
a solution cannot be found that satisfies the constraints on the variable 
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involved, the method hacks up and attempts to change the solution to the 
previous inequality. If it can, then the solution process proceeds. If 
not, it backtracks further in the attempt to find a sequence of solutions. 

If it hacks all the way, the system is unsolvahle. If all the inequalities 
in the path are solved, the system is solved, 

This general methodology produces a large number of test cases because of 
the boundary-interior approach to handling loops. Hence more recent 
research has been geared toward the selection of path segments of interest 
in order to restrict the number of test cases to be generated. This allows 
the examination of loops, specified sets of branches and/or loops without 
bearing the overhead of generating data for paths of peripheral interest. 

A command language has been introduced to enable the user to select the 
classes or subsets of classes of paths, A feature was added to include the 
identification of statements, loops and branches by number, allowing the 
user to communicate more easily with the system by naming paths and classes 
of paths through a program. Other features include user assertion capabilities 
for stating desired logical conditions and for assisting in the symbolic 
evaluation process. The DISSECT system which is now being experimented with 
represents the latest work in this area. 

McDonnell Douglas 
Dissect System . 

The DISSECT system can be used in different ways and can take on 
different forms, depending on the capabilities which have been 
implemented. In- using the system the programmer begins by preparing 
a number of cases . Each case describes an analysis to be carried out 
by the system. In its simplest form, a case consists of: 

(a) A procedure consisting of a sequence 6f cbmraahds which cause 
the selection from the program of a path, partial path or 
set of paths or partial paths. A path is defined to be any 
possible flow of control through the program, 

(b) A set of commands which directs the system to print out: 

Ci) a system of predicates which describes the data that 
causes the selected paths to be followed; and 
(ii) symbolic expressions which describe the cumulative 
effects of the computations carried out by the 
selected paths. 

Several of the possible more complex features have also been implemented. 
The more complex features can be used to carry out more sophisticated 
program analyses. The current system will allow a user to conditional.ly 
carry out path selection commands based on ifhether the selections Will 
cause the generation of inconsistent systems of predicates, to generate 
and insert assertions into the selected paths, end to manueOdy force 
certain simplifications of the pred.icates and output for selected paths* 

An extended system would allow the user to combine systems of predicates, 
symbolic output and assertions in verification conditions; to solve 
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systems of predicates to generate test data; and to prove correctness 
by proving verification conditions. Attempts to automate those last 
tvo features have not been completely successful in specially designed 
test data generation and proof of correctness systems and they are 
not planned for inclusion in DISSECT, 

In the following example the system is used to ’’read” a program, 

DISSECT can be used to help a user read a program by analyzing the 
program to see what computations are carried out by the program 
along selected paths and what data causes the paths to be selected. 

Each case which is preapred for input to the DISSECT system can contain 
a section of text which describes in English the subdomain of the program 
corresponding to the case and the actions carried out by the program 
over the subdomain for the case. The DISSECT system can he used to 
confirm that the selected paths corresponding to the case are applied 
in the situations and carry out the actions specified in the text for 
the case. In helping the user to read a prog*:’am, the system helps the 
user to confirm that the program implements its specifications. 

Example 

The program in Figure C«1 is called H)IV, The program comes from the 
13^ Scientific Subroutine Package and can be used to divide one 
polynomial by another. The input to PDIV consists of two vectors 
of polynomial coefficients of length IDIMX and IDIMY where IDIMX-1 
and IDIMY-l are the degrees of the polynomials. 

It is not immediately obvious from a casual reading of this program 
that it carries out division of polynomials. Moat of the difficulty 
is due to the language the program is written in and the style of 
programming. The program has been clearly designed to handle these 
classes of input: 

(i) Zero divisor: IDIMI = Oj 

(ii) Divisor of higher degree than dividend: IDIMY >IDIMX; and 

(iii) Divisor not of higher degree than dividend: IDIMY ^IDIMX, 

The actions taken hy the program for the first two classes of input 
are obvious. There is not need to construct DISSECT eases to analyze 
the actions and program subdomains for these classes, A user might 
still construct these cases in order to maintain, using DISSECT, a 
document containing the record of a complete exa^nation of all of the 
important cases covered, by PDIF, If so, he might construct the cases 
contained in Figure C-2 and Figure C-3, The case descriptions 
contained in Figures C-2 and C-3 include the output from the system. 

The user supplies the text description of the case and the DISSECT 
commands which specify a set of paths and the output to be generated 
for the paths. The system generates the systems of predicates 
and the symbolic output for the paths. If a case involves the 
specification of more than one path or partial path, the output for 
each path is grouped together under a SUBCASE heading. The output for 
CASE 1 in Figure C-2 involves two subcases. 
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SUBROUTINE PDIV(P,IDIMP »X,IDBIX, Y, 1 

1IDIMY,T0L,IER) 2 

DIMENSION P(l), X(l),Y(l) 3 

GALL PNORM (Y,IDIMY,rOL) k 

IF(IDIMY) 50,50,10 5 

10 IDIMP=IDIMX-IDIMY+1 6 

IF(IDIMP) 20,30,60 7 

G DIVISOR DEGREE TOO LARGE 8 

20 IDIMP=:0 9 

30 IER=0 10 

ho RETURN 11 

C ZERO DIVISOR 12 

50 IER=1 13 

GO TO ItO ih 

So IDIMX-IDIMY-1 15 

I«IDIMP 16 

70 II=I>t.IDIMX 17 

pCi)=x(ii)/y(idimy) 18 

G SUBTRACT MULTIPLE OF DIVISOR 19 

DO 80 K=1,IDIMX 20 

J=K-1+I 21 

X(J)=X(J)-P(I)»Y(K) 22 

60 CONTINUE 23 

1=1-1 2k 

IFCD 90,80,70 25 

C NORMALIZE REMAINDER POLYNOMIAL 26 

90 CALL PNORM(X,IDIMX,TOL) 27 

GO TO 30 28 

END 29 


Figure C-1, PD IV Program 
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CASE DmSOR TOO LARGE 
OUTPUT: PATE, PREDICATES . 

PATHS: DEFAULT SELECT ALL CONSISTENT; 

5 SELECT .GT.; 

T SELECT .LS, ; 

SUBCASE 1.1: 

PATH: 1-T, 9-11 

PREDICATES: 5 IDIMI .GT. 0 

T IDIMY-IDIMI+1 .LT 
11 RETURN 

SUBCASE 1.2: 

PATH: 1-T» 10, 11 

PREDICATES: 5 IDIMI .GT. 0 

T IDIMI-IDIMI+l . 
11 RETURN 


Figure G-2. CASE 1 — DIVISOR too large 


CASE 2 ZERO DIVISOE 
OUTPUT: PATE, PREDICATES. 

PATHS: DEFAULT SELECT ALL CONSISTENT 

5 SELECT .EQ. ; 

PATH: 1-5, 13, ill, 11, 

PREDICATES; 5 IDIMI .EQ. 0 
11 RETURN 


Figure C-3<. CASE 2 — Zero divisor 



Programs to be ayialyaed by DISSECT must have sequence numbers or line 
numbers. The text in the CASE section of CASE 1 describes the subset 
of the input domain consisting of pairs of polynomials where the 
divisor is of larger degree than the dividend. The path selection 
commands in the PATHS section describe the set of paths which the user 
claims to correspond to +-his case. The 5 SELECT ,GT, command causes 
the .GT, branch to be chosen in statement 5. The DEFAULT SELECT ALL 
command tells the system what to do if it encounters a conditional 
branching statement for which there is no associated SELECT command. 

The command instructs the system to follow all branches which do not 
cause the system. of predicates associated with the path that follows 
a branch to become inconsistent. The system analyses the program and 
produces the output in the SUBCASE sections. In more complicated cases 
the output will he useful in determining whether the program conforms 
to the program specifications in the CASE text descriptions. 

The action taken by the program for pairs of polynomials where the 
divisor is of degree less than or equal to the degree of the dividend 
is more obscure. The user may wish to confirm that the program works 
cox'rectly for this class of data by constructing cases for which: 
the divisor and the dividend both have minimal degree degree one; 
they both have the same non-minimal degree* say degree 2; and the 
divisor has smaller degree than the dividend* say divisor degree 2 and 
dividend degree 3. Figure C-4 contains the CASE corresponding to 
the output for which the divisor and the divident both have degree 2, Fig- 
ure C-5 contains the CASE corresponding to the input for which the 
divisor has degree 2 and the dividend degree 3, In both eases the user 
has requested that the system print out the path sequence munbers 
and the symbolic output for the paths. The ASSIGN command allows the 
user to give values to variables in paths. The result of the ASSIGN 
commands in CABE’s 3 and 4 is to cause the variable IDIi®! and IDIMY 
to be replaced with constants in the symbolic output for the CASES, 

The LOOP commands in CASES 3 and 4 specify how many times a loop in 
a path shoiild be executed. The command 25 SELECT ,GT,* ,LE.; specifies 
that the ,GT, branch in statement 25 should be followed during the 
first iteration of the loop containing the statement and the .LE, 
branch the second time. 

The system also allows explicit conditional commands. The conditions in 
these commands can involve "loop coimts", values of program variables* 
and path attributes. Path attributes are character strings which can 
be attached to any program statement. All paths which pass through 
a statement having an attribute acquire that attribute. 

The DISSECT system allows a user to isolate parts of his program and to 
apply automatic program analysis tools that will help him to see what 
circumstances^ cause those parts of the program to be executed. He can 
also apply analysis tools to see what computations are carried out by 
different parts of the program and to compare those with program specifi- 
cations, More complicated analysis tools can be incorporated into 
DISSECT to allow a user to ask for the automatic generation of test date, 
the formation of verification conditions and the proof of program properties. 
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CASE 3: X AI© X HAVE SAME DEGREE - 2 

OOTPOT; PAIDH, OUTPUT (P,X) 

PATHS: ASSIGN iDBfX “ 3; 

ASSIGN IDIMr ~ 3j 
5 SELECT ,GT,; 

7 SEIJSCT ,GT,i 

20 LOOP 2; 

25 SELECT «LE*; 

PA.TE; 1-7, 15“23, 20-23, 20, 24-28, 10, 11, 
OUTPUT: 

AREAI P: 

P(l) « X(3)/I(3) 

ARPAX X: 

XCl) - X(l) - CX(3)/X(3)) « Y(l) 

X(2) « X(2) - Cx(3)/YC3)) * y(2) 


Figure C-4. CASE 4 — EquaJ. Degrees 


CASE 4: X HAS GREATER DEGREE THAN Y - 3 AND 2. 

OUTPUT; PATH, 0UTFUT(P,X) 

PATHS; ASSIGN, IDIMX = 4; - . 

ASSIGN IDIMY = 3} 

5 SELECT ,GT,; 

7 SELECT ,GT,; 

20 LOOP 25 

25 SELECT ,GT., ,LE,j 

PATH: 1-7, 15“23, 20-23, 20, 24, 25, lT-23, 

20-23, 20, 24-28, 10, 11, 

OUTPUT: 

ARRAY P; 

P(l) - (XC3) - (x(4)/Y(3)) » Y{2))/Y(3) 

P(2) « X(4}/Y(3) 

ARRAY Xs 

XCl) = X(l) - (CX(3) - (x(4)/Y(3)) 

* Y(2))/Y(3)) * Y(i1 

XC2) ^ X(2) - (X(4)/Y(3)) ^ Y(l) - ((X(3) 

» (x(4)/YC3)) Y(2))/Y(3)) « Y( 2) 

X(3) = X(3) •» Cx(4)/Y(3)) * Y( 2) 


Figure C-5. CASE 5 — Degree of divisor less than degree of dividend 
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General Reseai'ch 


IT 

Miller has also developed a system that generates test cases for F0RQ?RA1T 
programs. The first step is the decomposition (if necessary) of the pro^am 
into a series of smaller se^aents that can be dealt vith separately. This 
process decomposes the program structure into segments vith th<i property 
that each has a single entry single exi.t. 

Miller’s approach is to construct a tree representation of the program hy 
automatically performing a series of reductions of the program, directed graph. 
This corresponds pretty much to Hovden’s reduction of the program to a 
state diagram of the class descriptions. 

Once the tree representation has been constructed, the structure (program 
control) is analysed to determine the program execution patterns (paths). 

This tool also uses the backtracking method to construct test cases. 

However, a basic difference between it and Howden’s method is the handling 
of loops and the extent of processing performed during backtracking, I'fhile 
Eowden derives his entire set of inequalities prior to the backtracking 
process, Miller performs much of the derivation during backtracking, Howden 
attempts to handle loops as part of the system to he solved, while Miller 
reduces loops to non-iterative flow* 

The backtracking in Miller’s system implements a set of rules that dictate 
the actions taken by the hackfcracker as it traverses the statement sequence 
in reverse order. The reduction of the program produces a set of simultaneous 
non-linear inequalities involving only input-space variables and constants. 
These inequalities are now solved and the result is the test case. 

In the case of iterative flow, manual intervention may ha necessary if a 
valid test case set cannot be constructed by a single traversal of the cycle. 

The approach to coverage is to exercise each predicate in the program at 
least once to each of its possible outcomes, hut not necessarily in every 
possible combination. 

The research at GRC in this area concentrated on tradeoffs between segment 
size and number, and in learning tZie effects in execution time requirements 
both in performing the computations and directing the hacktracker. 

TRW 

Hoffman^^*^^*^^*^^ of TRW, Houston, is developing the Automatic Test Data 
Generator (ATDG) for FORTRAH programs. This tool is currently capable of 
constructing paths through the unit of software and of identifying a 
characteristic set of paths required to exercise all logical decisions. 

The goal of ATDG is to exercise each transfer at least once using the 
fewest number of cases, 

Hoffman identifies all segments of a program by number, identifies all 
possible logical transfers between the segments and then defines a path 
through the software which is a chain of transfers beginning at an entry 
point and ending at an exit point of the unit of soft-i^are. 
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He accomplishes the paiH determination hy cons;tructing a directea.^:^ ■with 
each se^ent-:0£ ; the program^ r eprssei^ed as & node'; in the ^aph and. each 
logical transfer as a node connection, then perfoiming a netirork analysis to 
describe the pathSfl transfers' ^are’ the connections hei^ segments and 

a segment ends at. a predicate Vith the next segment . beginning:- the 
associated eKeCutable ]^t of the decision statement# 

Unexecutable paths are eliminated nsing: the, '’impassible pairs’’ teCto 
this technique identifies in^ossible pairs of tfansferSs based on data 
constraints that prevent a pair of transfers from 'being executed in the 
same pass through the soft'ware* 

In the majority of cases, the pairs are always impossibles However, if 
values of variables can "be changed, then it is necessary to qualify the 
possibility of transferring, 

the application of the impossible pairs technique to a short program 
illustrated in Cl?) reduced the number of possible x>aths from 3264 to 9. 
this -was further reduced to seven, to reduce the number of paths required 
to exercise all transfers, a connectivity matrix is generated representing 
the direct transfers in the software unit, A reachability matrix in then 
formed allowing a look-ahead capability to determine the best transfer to 
select based on the number of transfers which can be subsequently executed. 

Once executed, a transfer is flagged and each new path contains the largest 
number of n«w transfers, This process continues until all transfers are 
exercised, 

AEDG does not currently generate data to exercise the paths. The work is 
continuing and the expansion of the connectivity matrix concept is being 
pursued, The basic difference between Hoffman’s approach and that taken by 
Howden 'and Miller is the method of determining path construction, Hoffman 
defines the paths based on the data constraints, and displays paths which 
can be executed. This system is interactive with the user generating the 
test data which will exercise the path, 

Hoffman’s use of the software network analysis with the connectivity matrices 
allows the combination of the structural and logical characteristics of the 
unit to form a matrix defining executable paths based on data constraints. 

Loops are considered to require one iteration the first pass through the loop 
bumping the index and the second pass allowed to fall through. 

An ohviour, advantage in the concept of connectivity matrices is the ability 
to handle units of code that are complex. The siae of the matrices increases 
with the complexity of the program, but the network analysis and matrix 
generation and processing remains "the same. 

University of Colorado 

L, Clarke'^ of the University of Colorado describes the system developed there 
to generate test data for programs written in AHSI .FORTRftH, The system is an 
extension to the validation program, PAVIii^, which perfo'.ms data flow analysis. 


Programs aro deGomposed into segments, and a directed graph is created de- 
fining the possihle paths* A data haae is created containing information 
about the program units. This data is used during data generation. 

The lexical analysis performed creates a list of tokens subsequently trans- 
lated to an intermediate code similar to assembly language. This intermediate 
code in conjunction with the directed graph is accessed by the test data 
generator. An advantage to this approach is that adaptation to a new 
language would require a translation to the intermediate code, with only 
minor modifications necessary to create a test generation system for the 
new language, Howden uses a similar technique. 

This system addresses loops and subroutines calls. The user can designate 
the path to be taken through the program, requesting that loops be traversed 
a specified number of times and that the path enter and exit specified 
subroutines following a designated path. The control structure blocks 
defined by DAVE are numbered and stored in the data base. The paths are 
then described by the user in terms of the subroutine names and block numbers. 
Ms, Clarke offers the following exEuaple, A path is described by SUBl, 1,2,5, 
SUB2, 1,7,8, S0B1, (6,7) $2, EOPjEKD, An analysis of the path starts with 
subprogram SUBl. Blocks 1, 2 and 5 are symbolically executed, A call to 
subprogram SUB2 is encountered in block 5* Blocks 1, 7 and 8 in SUB2 are 
then executed, A return statement is s.icountered in block 8 of SUB2 and 
analysis returns to SUBl, block 5» The remaining code in block 5 is executed. 
Then the loop formed by blocks 6 and 7 's executed twice. 

The analysis determines the feasibility of the path. If it is found to be 
infeasible, the user is notified and analysis of the path is terminated. 

If the end of the path is encountered, the path is executable, therefore 
feasible, and the test data that drove the execution down that path is 
ret-urned to the user. 

Feasibility is determined by attempting the symbolic execution of a path. 

When a conditional branch is encountered a data constraint is generated. 

Each constraint is then passed to an inequality solver that attempts to find 
a solution to the set of constraints and confirm that they are consistent. 

If a solution exists, the symbolic execution of the path continues. If they 
are inconsistent, the user is notified and the path is considered infeasible. 
Linear progpramming techniques are used to solve the system, based on the 
premise that a large proportion of programs have linear constraints (allowing 
the use of the linear techniques) and non-linearities will not limit the use 
of the tools. 

This system operates also in the static mode, with a path being initially 
specified. 

Since the capability of traversing loops creates the possibility of an 
infinite number of paths, only a few of which may be of interest, this 
analysis program requires that the path to be analyzed be specified by 
the user. 


IBM 


20 

EFFIGY is an interactive symbolic executor developed by King 5 et.al, at IBM, 
It is applied to prograwa vritten in a special subset of PL/I containing only 
integer-valued variables and vectors, 

EFFIGY accepts one statement at a time, building a tree that defines the 
paths through the progrtaa as it goes along. At each branch, the user 
decides vhich path is of interest and communicates tliis to the system which 
then proceeds with the symbolic execxition. 

The path generation proceeds in a forward fashion as opposed to the back- 
trackirig methods used by Howden and Miller, 

EFFIGY saves the state of the branches and allows restoration to particular 
points so that alternative paths can be explored later# 

An interesting aspect of EFFIGY is the capability to accept assertions during 
the symbolic execution that allow comparison of the correctness specification 
to the data at a given point in the execution. 

The use of symbolic input data allows verification of the program over a 
range of numeric data without examining each specific possible input 
within that range. It represents another attempt to prove taht data of 
interest exists that will drive execution down a particular path, 

AuLor^tion of the process of exploring all paths of interest is being pursued 
by the developers. 

This system is limited in practical use, hut gives added insight into the 
concepts of symbolic execution as a tool in proving programs correct, 

Stanford Research Institute 
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SEIiEGT is a, system developed at Stanford Research Institute by Boyer, Elapas, 
and Levitt^l, It is also a symbolic executor, executing the code in a forward 
direction thi'ough the program. It is applied to programs that are written 
in a language that resembles a subset of LISP, since that was the language 
used as input to the SRI program verifier which was used as part of SELECT, 

As the statements are executed and branches are encountered, the path is 
generated and stored as a conjunction of predicates in a list. Variables 
are kept in another list which is updated whenever an assignment statement 
is encountered. Data that will drive execution down the path is also 
maintained and is derived by the execution of an inequality solver. 

As each predicate is examined, the inequality solver is called. It solves 
linear and some non-linear inequalities for both integer and real valued 
variables. If the inequality cannot be solved, the traversal of that path 
is abandoned and is tagged as an impossible path. 

Loops are addressed by SELECT as paths that are iterated a user-specified 
number of times. 
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SELECT also offers the capahility of accepting user-supplied assertions, They 
can he used to determine the numerical value of the symbolic data during 
execution, to constrain the input space hounds from vhich SELECT is to generate 
test data, or to provide a specification of the intent of the program to 
verify a given path. 

Calls to subroutines are presently handled by substituting the subroutine 
code into the program. This causes an explosion in the number of paths 
and is a limitation. Work is being done to address subroutine calls by 
characterising them by input and output assertions, and replacing the calls 
by these specifications. 

All teat case generators currently described are experimental tools* The 
various techniques employed in developing these tools represent attempts 
to achieve a feasible testing aid. All require total knowledge of the 
program structure as well as the intent, and all are severely limited in 
the size of the program that can be accepted. 

The capability of DISSECT, SELECT and EFFIGY to accept assertions is an added 
step toward proving the correctness of a program. It allows the programmer 
to express his testing requirements in some way that reflects the program's 
intent, and to compare this intent with the results of data derived by the 
structural analysis of the program. 

Execution Analyzers 

General Overview 

The basic function of execution analyzers is to gather run time statistics 
that can give a programmer insight into the behavior of this program. 

Initial attempts to analyze program behavior were through the use of trace 
functions. While tracing indeed shows the order and state of execution, 
it is very costly in programs that are not small, and particularly if loops 
are involved, Tracing is still, a valuable debugging aid since it provides 
for a display of the contents of cells of interest during execution, 

A number of effective execution analyzers have been developed to perform 
dynamic analysis of the code through the use of external performance 
monitoring techniques or with software probes inserted into the source code, 

Boole and Babbage’s Problem Pirogram Evaluator. (PPE) is an example of 
a tool that monitors execution performance externally, PPE is linked to 
the load module and causes execution to be interrupted periodically, 
recording the state of the program’s execution at each interrupt. The 
information is compiled and displayed graphically. While this technique does 
not disturb the test object code and is language independent because it 
operates at the object code level, it does not provide complete statistics 
about execution frequency, particiiLarly in tight loops. 

The analyzers that insert probes into the source code are language dependent, 
Among the currently available tools developed for the analysis of FORTHAIJ 
programs are MDAC’s Program Evaluator and Tester .PET), TRW’s Flow program 
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vhich is one of the tools in their Product Assurance Confidence Evaluator 
System (PACE), CSA's QUALIFIER, GRC^s RXVP, CAPEX's FORTUNE, and the NBS 
Analyzer, CO^L analyzers include ADH*s Meta Cohol, NCI's Series-J, 
and CAPEX's COTUNE, MDAC has developed a system to analyze the execution of 
TACPOL/MOL programs for the Army at Port Belvoir and is currently vorking 
on a prototype PL/I analyzer, and GRC is developing a JOVIAL analyzer for RADC, 

Several techniques have heen used to accomplish instrumentation of source code. 
All are based upon the recognition of the program's control structure • 

Entry points, exit points, branches and loops must be recognized to determine 
■where the probes should be inserted to give the desired results. Most of the 
tools provide a control language that allo'ws the programmer to communicate 
■with the tool expressing his areas of interest, such as instrumenting only 
selected code segments and requesting optional statistics. The control 
langu^e may be transparent to the compiler (in the fora of comments) or 
may be implemented as additional verbs ■which must he removed after testing 
is completed. 

In either case, the source code, with any embedded control statements is 
passed through a preprocessor. The control statements direct the instru- 
menting process (in most cases, default conditions are used when no control 
statements are present). The preprocessor generates a modified version of 
the source code that contains the instrumentation. The probes are either 
calls to subroutines in -which the statistics gathering takes place or are 
Goxinters in “the form of assignment statements that collect the statistics 
in arrays and symbol tables. The performance of the instrumented code is 
degraded by the added code, but the functional results remain the same 
(i.e, the program still does -what it did before it -was instrumented). 

Descriptions of the individual tools along -with performance statistics 
for each ■were gathered during this study (see Appendix A). 

All of these tools currently do not actually verify the program. Their 
value lies in the insight they provide to the programmer about the behaviour 
of the code in relation to its structure. Dead code can be located, 
impossible branches can be identified, and high utilization code can be 
isolated and optimized. 

An initial attempt oo address the function as well as the structure of 
the code -was proposed by Stucki and Foshee^^ as an extension to PET, MDAC’s 
execution analyzer, A prototype PL/I implementation Is currently being 
tested. 

As PET is now implemented, it provides information about the variables in 
the program as it was executed; namely minimum and maximum values assigned 
to the ‘Variables at each assignment and for each do-loop index variable, 
and first and last values for variables in each assignment statement. 
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Current Research 


Assertion Checking - General Philosophy 

The assertion concepts for programming languages which are now being 
developed constitute major extensions to our ability to carry out 
"systematic progra mm ing" , These new assertion concepts will impact 
all phases of the software life cycle from initial requirements and 
design phases down through certification, and maintenance iterations. 
These assertion concepts are designed to encourage the development 
of algorithmic validation criteria as the implementation evolves 
fr^m the initial algorithm requirements and specifications down 
CO the final program code. 

The impact of these assertions will be both psychological and real. 

They will be "psychological" in the sense that they are omnipresent. 
Embedded as comments within the program code, they will be a positive 
influence for increased understanding and awareness of how our 
algoi'ithms should behave and how we plan to insure that our algorithms 
do in actuality behave properly at all times. The impact of the 
assertions will be "real" in that they can be automatically checked 
and monitored under dynamic program execution. 

The assertion capability will allow programmers to verify to some 
extent that the code performs properly within the constraints 
defined by the assertions. Critical parameters can he dynamically 
checked or monitored for range, value, and order violations based 
on the prescribed bounds of the assertions. Subscripts are checked 
for range violations, and given subroutine parameters are checked 
for change of value during execution of the subroutine. 

The inclusion of specifications data in the program now begins to relate 
the code to the requirements and is a significant advance toward 
verifying the program. Module interfaces can he examined via the 
calling parameter assertions, providing added confidence in the 
integration of modules. 

An important side effect of the assertion capability is the dociamen- 
tation in the code of the critical requirements of the program by 
the assertion statements. This enhances the understanding and 
maintenance of the program at all levels of development and operation. 

Application of these techniques and their associated tools offers a 
positive step forward in the development of more reliable software 
systems. This approach can be applied to existing programming 
languages today via extensions to currently existing automated tools 
like MDAC’s PET system. The approach also promises to impact future 
language and language processor design. 


Basic Fro;pertieB of Assertions 


The premise upon which the assertion concepts is hased. is the need we 
have for thinhing through and thoroughly understanding the expected and 
actual behavior of algorithms. The emphasis will not be placed on 
proving mathematical or logical properties of programs but rather on 
an attempt to increase our understanding of the nature and behavior 
of the algorithms we use. It is openly acknowledged that some purists 
(i,e,, London, Good, et.al, ) may feel that we are taking a rather 
informal approach to the study of program properties. However, it should 
be noted that the assertion language which will be developed contains 
sufficient power to serve as a vehicle for stating many formal properties 
of algorithms,^^ Indeed, at some future point in time a theorem 
proving tool may well interface with our embedded assertion language. 

The assertion language will address both our imderstanding of flow of 
control and flow of data through algorithms. A hierarchy of assertion 
cor struct s will be defined to make their use more natural and convenient. 
It should he noted, however, that one assertion construct is really 
sufficient (i»e,, a generalised local assertion). 

Generalised Local Assertion 


A generalised local assertion may be embedded in a comment at any point 
within the executable code of a program where another executable 
statement may appear. The local assertion is designed to enhance the 
documentation of critical algorithms throughout the entire life cycle 
of the software. Dynamic execution time checks can be activated at 
selected points in time to ensure that the actual run-time environment 
is consistent with the logical state specified in the assertion. This 
dynamic assertion checking can be used to great benefit in debugging, 
validation, certification, and maintenance of complex systems. 

The format of the generalized local assertion isi 

ASSERT L0CAL(exbended-logical-e3q>ression) [optional-qualifiers] 

[ control-options ] 


The exact placement and treatment of the assertions will be tailored 
to the existing language facilities in currently defined languages. 

In these currently available languages the assertions will be mplemented 
through specialized comments processed by a source code preprocessor. 

New language development and futiire compilers for existing languages 
may contain options for directly implementing the assertions. 


Optional-Qualifiers 


In order to provide an existential and universal qualifier notion to 
the generalized local assertion an optional looping capability is 


defined: ...FOR 


ALL 

SOME 


II 


^[[variable-list) 


[set of ranges/values)] 


WHERE 


Ctjuantifier-r-eontrolling-logical-exiir ) . 
e.g, C ASSm CxClh== X(J)| FOH ALL (X;&X WHm 

means: C^I,J s.t, J^8A I ^ ^ X(I)54xCJ) 

Assertion Control Options 

The total control alluded to above (i.e. , ignoring all assertions by- 
treating them as comments) offers the user a binary choice as to 
whether or not to apply dynamic assertions during program development j 
however, other levels of control are provided within the assertion 
language itself. 

The assertion language itslef contains three hierarchical levels 
of control; 

1) instrumentation control - control of those sets of assertions 
which will he instrumented at a given level of testing, 

2) dynamic control - run-time control of those instrumented 
assertions which are to he dynamiclaly checked, and 

3) threshold control - user control when assertion violations 
are ohserved. 

Instrumentation control is provided by a L3SVEL option. The syntax 
of this option is; 

, , .LEVEL ( pr eprocessor-c ontr ol-expr ess ion ) 

The LEVEL option provides control information to the preprocessor telling 
it which sets of assertions should he considered for dynamic analysis. 
This level of control provides a means for testing selected software 
features at various points within the software development cycle and 
fits in well with the top do-wn approach to program development. This 
also allows a user to group sets of assertions together for various 
types of dynamic checks. 

Dynamic control is provided by a condition option. The syntax of this 
option is; 

. , .CONDITION (dynamic-control-expression) 

The CONDITION option provides run-time control of the assertions which 
have been built into the program. This option affects only those 
assertions which have been actually instrumented, thus the CONDITION 
option is of lower priority than the LEVEL option. It should also he 
noted that the COHDITION option can he dynamically changed under 
program control to activate or deactivate the assertion as often as 
desired* 








/ i 
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Threshold control is provided by a LIMIT option. The syntax of tliin 
option is: 

LIMIT n{VIOLATIOHSj I] 

llEXlT [VIA] proc~namei I 

The LIMIT option provides user control in the event of n violations of 
the corresponding assertion. The user can specify that control being 
transferred to a ‘wrap-up procedure proc-name if the EXIT phrase is 
specified. Otherwise, the HALT phrase will simply terminate execution 
and generated an assertion report automatically if n assertion vielations 
are encountered. Motivated by a need to make assertions about arrays 
as well as scalarn, the following notation has been adopted. 

Array dotation for Assertions 

Two areas of concern immediately arise when discussing data arrays, namely, 
array indices and array values. Thus, if one is monitoring program 
behavior, it is not enough to monitor array values alone, since program 
logic is invariably concerned with where these values are stored within the 
arra> , 

The approach is to generalize the assertion and monitor capabilities 
to include data arrays. Array notation is as follows: 

Assume an array of the form . . . ^n) , References to specific 

subsets of array values or array indices are indicated by A(l 2 ,Ig,l 3 , . , , 
Ijj), where is a subrange of Ij^. This notation is position 
dependent; i.e., if Ip is not referenced, its position must be indicated 
by an asterisk (*), as in A (l^j^I^, . .Ij^) , The format of each is 
S,;y where (see note below). If £.=y £;y may be replaced by y, 

as A{£^ :y^,yj^,.,,). Thus for A(10,20) we might reference 

A(5, 10:15) 

A(*,3) 

A(2:5,») 

A(2;6,2:10) 


Extended Logical Expressions 

Two types of extended local operations have initially been defirjed for 
the assertion language. An array to scalar logical operation will be 
allowed ■> ith its result being defined as 'true' if and only if ail 
component to scalar operations are found to be true. An array to 
array logical operation will be allowed for identically specified array 
cross*»sections with its result being defined to be true if and only if 
each pairwise component operation yields a result of true. 

Note; If the character (colon) is not available on a apeeiric 
machine, another suitable character can be substituted. 
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Local Assertion Examples 


A simple local assertion example is sho-wn "belov in a typical report 

format. The assertion simply indicates that at the point vhei'e it is inserted 

into the source code ve expect the value of the variable MOVE to be 

less than 9- The report format indicates that this assertion was 

checked 9 times. Violations were noted on the 6th and Ttb executions 

of the assertion. It is furthermore noted that MOVfcl actually contained 

the value 9 on those two instances. A snapshot is taken of all 

pertinent variable veilues associated with the violation when the 

trace mode is specified. 

EXECUTION 
COUNT 


00045 C ASSERT LOCAL (MOVE .LT, 


1 

9) LIMIT 10 9 


SPECIFIC EXECUTION DATA 


ASSERTION VIOLATIONS 


tjii 


EXEC HUMBER 
6 
7 


VARIABLE VALUE 
MOVE 9 

MOVE 9 


It is also worth noting that had we encountered 10 violations we would 
have halted execution at this point in the program. 

Examples of the use of array cross-sections in extended logical expressions 
include the following: (assume an array A(l0,20) has been defined) 

(a) ASSERT LOCAL (A(*,3) .LT. lO) 

LIMIT 6 VIOLATIONS 

(b) ASSERT L0CAL(A(?:6,2:10) .HE. O) 

(c) assert local (A(«,*) ,GT. 0) 

In (a), the value of each array element whose second subscript^! is 
checked and reported as a violation if its value is not less than 10, 

Ten array values will be checked in all. Any number of assertion violations 
within an array operation cause the operation to be counted as a single 
assertion violation. Thus, the LIMIT 6 VIOLATIONS is concerned with 
only the number of invalid operations not with the niunber of violations 
within the array. 

In (b), only array values within the specified subscript ranges are 
checked for an assertion violation. In (c) all array values are 
checked for an assertion violation. 

Specialized Local Assertions 

A number of additional specialized local assertions are proposed to 
facilitate the expression of user validation criteria. This exrtiensihle 
attribute of the local assertion concept is illustrated by the 
following constructs: 
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T 
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ASSERT LOCAL VALUE[S] (variable-list ) (set-of~legal ranges /values) 

[ control-options ] 

ASSERT LOCAL VALUE[S] (variable-list) NOT (set-of-illegal ranges/values) . 

ASSERT LOCAL VALUE[S] (variable-list) INVARIANT... 

ASSERT LOCAL SUBSCRIPT RANGE (list-of-array-specification . 

ASSERT LOCAL ORDER (array-crosB— sect ion) |[ ASCENDING ||***^ 

i DESCENDING! I 


All of these specialized local assertions could be replaced by one or 
more generalised local assertions, however, their existence facilitates 
the graceful transition fVom program requirements and their associated 
validation criteria to embedded program documentation in the evolving 
code. 


The first three constructs cause instrumentation to be generated at the 
position where they occur. The latter two constructs specify that 
the next executable statement will respectively be checked to insure 
that it does not alter the value of an invariant variable (e.g., ' 
through side effects from subroutine or function calls) or use subscripts 
outside the specified ranges. 

The ASSERT ORDER statement checks a sequence of array values as follows: 

ASSERT LOCAL ORDER (A(«,3)) ASCENDING 

For an array A(10,20), the following assertion violation summary 
illustrates the type of information traced for a violation: 

EXECUTION 

COUNT SPECIFIC EXECUTION DATA 


229 ASSERTION VIOLATIONS 1 

EXEC NUMBER SEQUENCE SNAPSHOT VALUE 
18 A(Y,3) 6 

A(8,3) 100* 

A(9,3) 8 


The ASSERT VALUES statement checks variable values against a specific 
set of ranges/values. The ASSERT SUBSCRIPT RANGE statement will 
check addressing on those arrays specified to ensure that only those 
portions of the array specifically selected are accessed. A subsequent 
example will illustrate the usefulness of this concept later in this 
paper. All of these latter constructions will i*esult in providing 
similar traces to those already presented for out of bound conditions. 


Tlie Concept of a Global Assertion. 

Expanding our notion of assertions, ve iamediately identify the need 
to expand to scope of application for our asserted program properties. 
In an effort to avoid requiring several similar local assertions within 
a particular program region, the concept of a glohal assertion has 
been introduced. This is a novel approach which promises to have a 
significant impact on the way we design, implement, and test software. 

Global assertions will allow us to extend our capacity to inspect 
certain behavioral patterns for entire program modules, selected 
regions of modules or module interfaces (entries and/or exits). 

Global assei'tions appear in the declaration section of the 
program module. 

These glohal assertions will have effect within the scope defined 
(i.e,, globally at all pertinent points, regionally over the named 
region, collectively for all entries, and/or collectively for all 
exits.) 

The YMjUES statement inspects each specified variable as its value 
changes and reports when: (option l) the new value is not one 

of the specified legal ranges/values, or (option 2) the new value 
assumes a specified illegal ranges/values, or (option 3) checks to 
make sure the values of the selected variables are preserved 
(i.e., no direct or externally caused changes are permitted). 

The ASSERT SUBSCRIPT RANGE statement -\rerifies that array subscripts 
fall within a specified range wheneveT' the array s referenced during 
program execution. It should he noted that this statement provides 
a meajis of checking portions of aj/rays as well as normal upper and 
lower bounds. For this reason, it is more powerful than the PL/I 
type ON SUBSCRIPT EAl^’GE check. 

Instrumentation will be inserted into the source program by the 
preprc. essor to accumulate the following statistics relative to 
assertion violations; 

(1) Identify the statement that caused the assertion violation. 
For that statement an execution count and violation 
execution counts identical to those obtained for local 
assertions are reported, 

( 2 ) The actimuL value that caused the violation. This value 
is lixiked to the statistics identified in (l) above. 


Some FOKTRAM examples follow: 


FP 

\k 






(T 

i 

b>.: 

UJ 


i 


V 


20 


DIMENSION Ado, 20) 

21 

C 

GLOBAL TRACE 10 VIOLATIONS 

22 

C 

ASSERT VALUES ( I, J,K,L) (0:100) 

23 

c 

ASSERT VAIjUSS(II,LL) (-10:10) 

24 

c 

ASSERT VAiUES (KK,NN) (2,4,6,8,10) 

25 

c 

ASSERT SUBSCRIPT RANGE (A(*,31)) 

26 

c 

ASSERT VALUES (X,Y,Z) INVARIANT 


rfi>» 

1 1 


102 

103 


K = K + 1 

II = A(L,J) ^ LL 


P 


234 

K - A(J,K) t 1*100 

235 

II = II + 2 

236 

NN = KK«(I-J) 






300 


CALL ROUTINEX(X,Y) 







P 

II 



1$ 


If assertion violations occurred in this example, the following 
statistics are indicative of what would he reported by the Postprocessor: 


102 


103 


Execution 

Annotated Frof^ram Listing Count 
K = K + 1 511 


II = aCL,J) + LL 511 


Specific Execution Data 

ASSERTION VIOLATIONS 1 

ASSERT (I,J,K,L) (0,100) 

EXEC NUMBER VALUE 
10 101 


ASSERTION VIOLATIONS : 

ASSERT (II,LL) (-10,10) 

EXEC NUMBER VALUE 
22 20 

ASSERT SUBSCRIPT RANGE(A(« ,3) ) 
EXEC NUMBER VALUE 
5 A(12,3) 

105 A(l,4) 


s 
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ASSERTIOfT VIDLATIOHS k 

ASSERT (I,J,K,L) (0,100) 

EXEC iraMBER VALUE 

52 101 

53 102 
ASSERT SUBSCRIPT RAKGE(A(* ,3) ) 

EXEC NUMBER VALUE 

52 A( 5 »U) 

53 A(6,U) 

ASSERTION VIOLATIONS 1 

ASSERT VALUE (II,LL) (-10,10) 

EXEC NUMBER VALUE 

50 12 

ASSERTION VIOLATIONS 1 

ASSERT VALUES (KK,NH) (2,lt,6,8,10) 
EXEC 1R3MBER VALUE 

20 7 

ASSERTION VIOLATIONS 1 

ASSERT VALUES (X,Y,Z) INVARIANT 

VALUE OF CALL FARM X 
EXEC NUMBER BEFORE CALL AFTER GALL 

30 -10 -20 


Structural and Static Analyzers 

Structural analyzers are tools that examine the program code, performing 
an, analysis of the structure. Execution of the code is not required 
for this analysis. 

The analysis resiilts in definition of the internal control structure 
describing the paths through the program and can he depicted as a 
directed graph. Other analyses can he performed based on the decomposition 
of the structure into a tree representation. 

Structural analysis is performed hy JOYCE(MDAC), RXVP(GRC)^^, PACE(TRW)^^ 
and both HRNANL and DAVE (University of Colorado )^9j2 5^ These systems 
all provide euci error finding capability at this level, and in addition, 
are the foundations of the test case generators developed by each organization. 
They all are designed for analysis of FORTRAN programs, 

Hoffman's Automated Test Data Generator (ATDG) uses the structural analyzer 
component for PACE, Miller's Automated Test Case Generator uses the 
structural analyzer component of RXVP, Clarke's test case generator uses 
the structural analyzer, DAVE, The decomposition methodology is described 
in the discussions of the respective test case generators. 


234 K w A(J,K) + 1*100 125 


235 II - II + 2 125 


236 NN = (X»(I-J) 38 


300 CALL E0UTINEX(X,Y) 53 


The analysis of a program’s structure also provides a foundation for 
various types of static analysis* particularly the relationships and 
data flows within programs, 

DAVE examines a program consisting of one or more routines , and checks 
for a numher of common errors not detected hy compilers. Its philosophy 
is "based on two rules expected to be o"beyed in the execution of a program, 

1} that a varia"ble is not referenced unless previously defined, and 2) 
that once defined, it is su"bsequ.ently referenced, "before "being redefined or 
undefined. The principle here is that many common: programming errors caUiie 
these rules to "be violated. Therefore, a seaPch for violation of the riilns 
should reveal the errors of possi"bility for errors that cause the violaticns. 

Among the errors that can cause Violations of the rules are uninitialized 
variables, misspelling of variable names, unequal lengths of corresponding 
argument and parameter lists, and mismatched types and diraerisions of 
arguments and parameters, 

DAVE constructs a call graph which represents the relationships between 
subprograms being called. The subprograms that do not invoke other sub- 
programs are called leaf subprograms and analysis of the code begins with 
them. 

In order to detect the rule" violations, it is necessary to know the usage 
of every variable in every statement, (i, a, whether it is used as input 
or output for each case). Therefore, a search of the subprograms is 
conducted along the paths defined by the structural analysis, to look 
for the rule violations, DAVE has addressed the problem of data passing 
through calling parameters and through COMMOh, and provides information 
about these variables in terms of their input/output classification. 

Much of the analysis performed by DA"VE is not designed to identify specific 
errors but to sense suspicious situations where errors commonly lurk, 
and to pass Warnings to the user who must then determine whether or not 
an error actually exists, or whether the program can he improved. 

The static analysis provided by EXVP provides much the same capability, but 
with added featin:es such as statement classification by type of statement, 
number in each module and percentage of total {note that this is also a 
capability of the execution analyzers); a module cross reference table of 
variables, the statements that reference them, and their type of usage; 
and an array subscript check for indexing appropriate to the array definition 

TRW’s PACE program incorporates some static analysis capabilities into its 
structural, analysis and execution analysis program, 

JOYCE is an automat ic checkout and documentation aid. It compiles tables 
of symbols and cross references of symbol usage within each routine of a 
program. These symbols include FORTRAN variable names the names of any 
referenced function or module, any entry points, and all I/O file 
references^ JOYCE permits the Input of symbol descriptions as data. This 
information may describe or designate a variable definition, a math s:/mhoi, 
flags for grouping related subsets, or subroutine usage information. 


OEIGMAL PAGE IS 
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The edited information may he combined to produce several combinations of 
descriptive reports. 

The cross-reference lists are useful for verifying consistency of symbol 
naming and usage, for finding typographical erroi's in coding and checking 
a program’s logic flow. 

Subroutine flowlists aid in verifying the accuracy of the modules logic 
flow. 

It is impossible to separate the tools strictly by function. There is a 

great deal of overlap in the types of analysis performed by each tool. 

Seldom does a tool serve one function, therefox'e, it is difficult to 

evaluate or even discuss them solely within one given category. For 

example, MDAC’s PET program, while basically classified as an execution 

analyzer, also performs static analysis in the form of providing a syntactic 

profile giving the statement types and nimiber of each. Again, the syntactic 

profile does not in itself identify specific errors, but provides information 

usefxil in verifying that certain properties of the program exist (e.g, 

ample coramentary) or that certain violations do not exist (e.g. non- 

standarii statements), 

% 

Test File Managers and Generators 

A teat file generator diffei*s from a test case generator in that it generates 
data based solely on input parameters specified by the programmer rather 
than on any knowledge of the program. Its purpose is not to try to prove 
anything about a program but to relieve the programmer (or test personnel) 
of the tedium of manually generating large volumes of data. 

Test file managers are tools that allow easy manipulation of test files 
once they are generated. 

These tools are commercially available for COBOL applications where they 
are particularly useful in testing systems that are data base driven. Among 
those offered are PRO/TEST (Synergetics )2^, Series-J (NCl)^^, and MetaCOBOL 
(ADR) 28. 

PRO/TEST is a set of three tools, the Test Data Generator, File PRocessor 
and File Checker, 

The test data generator generates data based on parameters specified on 
input cards. These parameters specify the file structure (record and 
field definition), the data ranges, conditional operations such as generation 
based on comparisons, and computational operations which derive data by 
operating on data from two data fields or one data field and a constant. 

Random specification always causes generation of the same data from the same 
parameters. 

The PRO/TEST file processor provides the capability of coupling live data 
with generated data, allowing selection and editing of records. 

The File Checker matches the output of a program to the original record 
design specifications to verify that the format and the structure are correct. 
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5ei‘ies-J offers a test data generation capability by parameter specification 
internally. It provides a Test Division in the code that specifies the data 
and its format. It then generates programs that will generate the data from 
tables within- the generated programs . 

The test data generator generates data based on parameters specified on input 
cards. These parameters specify the file structure (record and field 
definition), the data ranges, conditional operations such as generation 
based on comparisons, and computational operations which derive data by 
operating on data from two data fields or one data field and a constant. Random 
specification always causes generation of the same data from the same parameters. 
« 

The PRO/TEST File Processor provides the capability of coupling live data ivith 
generated data, allowing selection and editing of records. 

The File Checker matches the output of a program to the original record design 
specifications to verify that the format and structure are correct. 

The MetaCOBOL system includes a test data generator that also accepts input 
parameter specifications, generating data as specified. 

Other tools of this type include tiDAC’s random data generator which uses the 
computer's internal clock as the seed for the random number generator, and 
then builds a test data file within the bounds of input parameter specifications. 
This tool generates numeric data as opposed to structural data generally 
output by the COBOL data generators, and is currently used to test FORTRAK 
programs . 

TRV/ has two tools called ATC and RETEST that are test file managers for 
FORTRAN systems. ATC provides the capability to store pre-defined test 
cases in the data base, edit these test cases, and selectively execute the 
test cases. It also assists in the automatic comparison of test output 
against previously generated output. 

RETEST is a tool used to identify test cases required to reverify software 
that was previously verified and to identify new cases where required because 
of coding changes. 

CSA offers commercially a tool named RETEST which is similar to TRW’s RETEST. 
Miscellaneous Tools 


There are a number of other tools that <:an aid in program testing. Most of 
these are application dependent and must be redeveloped for every 
application. 


Ho-wever, there are a few that provide valuable information and are not appli- 
cation-dependent* MDAG's JOYCE Is an example. 

Probably the most commonly used type of tool is a simulator. While there 
are myriad simulators in existence, the requirements upon them are usually 
highly specialiaed, Therefore, there are not any generalized simulators 
in common use. The two simulators offered commercially, SAM (ADR) and CASE 
(TESDATA) are management visibility tools rather than actual test tools, 
and therefore will not he discussed here. 

There are many types of tools which have a common concept behind their design 
but which are application dependent due to the need to use data specifically 
unique to the system being tested. These tools include data verifiers, event 
loggers and their associated deloggers, and performance analyzers. 

There are several tools available whose utility is strictly in the debugging 
area. While there is often a fine line between debugging and development 
testing, this study will not address those tools that can he defined as 
debugging aids. 

« 

ConcluaionB 


This task, stirveyed many of the currently available program testing tools from 
the viewpoint of philosophy as opposed to performance which was addressed 
in Task I (see Appendix A)» 

The application-independent tools in use at present do not verify that 
the code in any way meets the specifications for which it was written. They 
give insight into the structure and behavior of the code. However, in 
light of the size and complexity of today’s computerized systems, any 
tool that relieves the programmer and the tester of time-consuming, 
tedius, and error-prone work is a valuable asset in the production of 
reliable software, . 

One new and very promising area of research now being explored addresses 
techniques for using tools to check functional attributes of programs. 

Through the dynamic checking of asserted program properties testing tools 

can provide valuable feedback on program compliance with selected specifications. 

As research continues it appears that the concept of proving that a program 
reliably solves the problem that was intended will eventually become a 
reality. In today’s environment, where practicality is a prime issue, the 
tools that can provide the most useful help to the user are the standards 
checkers, the execution analyzers (provided the language and machine 
attributes match), the test data generators and managers, and the structural 
analyzers. Other tools such as test case generators like DISSECT are 
still in the experimental stage of development and will require more research 
before they reach the stage of practical usefulness. 
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In surveying "these tools, the most common complaint was the difficulty of 
use. As new tools are developed and old ones are upgraded, greater attention 
to user orientation will help make these tools more valuable, simply because 
they will be used more. 
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Appendix D 

STBUCTURED PROGRAMMING AND PROGRAM MANAGEMENT TECHNIQUES 
D,X INTRODUCTION TO STRUCTURED PROGRAMMING 

The phrftse '’structured progranjiulng” prohahly had its origin in Dijlc 3 tra*s 
’’Notes on Structured Prograanning" vhlch were privately circulated prior to 
heing published [Reference 1], In these "Notes", Dijkstra's main concerns 
are the problems of very large programs and the methods by which their 
reliability can be improved. At least three major ideas are present* 

1, There is a need for some sort of a "demonstration of a program's 
correctness” to supplement the standard functions of design 
code and teat, 

2, Programs should be coded using only three types of control 
structures , 

3* Programs should be composed in a top-down manner utilizing 
systematic design techniques. 

Taieh together, these ideas are revolutionary even though all three were 
developed to some degree prior to Dijkstra's Notes. His major contribution 
was to bring these three diverse ideas together with a convincing argument 
that only in this manner could reliable software be developed. 

Dijkstra apparently feels that there is a deficiency in the standard cycle 
of design, code and test and that some sort of "demonstration of a programs 
correctness" should be included in this cycle in the future * He points 
out that program testing is an imperfect answer to the problem since 
"Program testing can be used to show the presence of bugs, but never their 
absence". This idea of demonstrating program correctness is being seriously 
considered in the more practical oriented segments of the computer industry 
but to date it has been primarily a subject for research and development. 

In its pure form, the method of "demonstrating program correctness" which 
has evolved, is very similar to the proof of a theorem in theoretical 
mathematics. Both manual and automated methods of performing such "'orogram 
proving" have been studied intensively, 

A pertinent question at the present stage of development is whether some 
form of useful demonstration of program correctness is possible short of 
the very sophisticated techniques of program proving. For exai^iple, a 
reasonably rigorous discussion of key facets of an cULgorithms' mathematical 
and/or logical basis could easily be imposed as a minimum requirement . 

Such information is often available in a document describing the algorithm 
development but usually this information does not impact the coding phase 


D-1 


and more importantly, does not form the "basis for any system testing. 
Perhaps algorithm code should be designed not for maximum efficiency but 
rather for maximum clarity of these key mathematical and/or logical bases. 
The results of research in the program proving field suggest examples of 
the types of things which should perhaps be emphasized in both coding and 
testing. 

Non~T:rivial Loops 

Some reassurance should he given that loops terminate properly under 
all possible conditions. The parameters that control looping should 
be identified and the exact manner in which these parameters change 
should he explicitly stated. Some reassurance should he given that 
the loop will he executed the correct number of times. 

Invariants 


An invariant is a relation which is true at every iteration of a 
loop, A clear and concise demonstration that a loop truly 
acccanplishes what it is designed to do is often most easily performed 
by identifying an invariant associated with it. [Reference 2 demon- 
strates the use of invariants]. 

Decision Structures 


Some reassurance should be given that all possible alternatives have 
been accounted for by some path in the decision structxire. The 
conditions under which a particular program path will be taken should 
be explicitly stated. 

The key feature seems to he explicit identification of key parameters and 
relations. Once this has been accomplished, automated methods can be used 
to verify that these assertions are valid. For exaa^le, software probes 
can be inserted in the program to be tested to verify that loops are 
executed the correct number of times, that invariant relationships always 
remain true, and that a deliberately designed series of tests exercise 
all paths of a decision structure. 

The idea that programs should he coded using only three types of control 
structures probably dates to a paper which proves a theorem concerning 
the flowcharts of proper programs. [Reference 3] (A program - or program 
se^nent - is proper if it has a single entry point and a single exit point ) . 
The theorem states in essence that the flowchart of every proper program 
can be represented by an ecjuivalent flowchart which is composed of only 
three basic control structtires (Figure 1). Dijkstra goes a step further 
and suggests that only these three control stsmactures should be used. 

[In Reference k, be singles out the GOTO or unconditional branch instruc- 
tion as a particularly common offender of this coding methodology). 

In support of this suggestion, Dijkstra presents several compelling 
arguments; 


Cl) The intellectual effort necessary to understand such a structured 
program is roughly proportional to its length (ineasxired, in some 
loose sense). This is not true of unstructured programs vhose 
complexity often increases geometrically with length. 

(2) The SEQUENCE and IF-THEN-ELSE control structures can be under- 
stood by enumerative reasoning and the DO-WHILE control 
structure by mathematical induction. Because "we know the 
appropriate pattern of reasoning";, the task of demonstrating a 
program’s correctness is made easier, 

(3) In such a structured program, the machine’s "progress through 
the computation is mapped on progress through the text in 

the most straightforward manner," In other words, the program’s 
execution sequence is more like the instruction sequence written 
on the listing than for a non-structured program. 

The third ma,jor idea in Dljkstra’s "Notes" is the concept of what has come 
to be called top-down programmin ;5 (Dijkstra called this Stepwise Program 
Composition), As motivation for composing programs in this manner, Dijkstra 
cites the problems of program modification and program manageability. 

Other authors including Wirth have also discussed this topic extensively 
[Reference 5). 

The basic approach is to compose a program in minute steps, deciding at each 
step as little as possible. As the problem analysis proceeds, so does the 
further refinement of the program. In such a stepwise approach, certain 
aspects of the problem statement are ignored at the beginning. This 
judicious postponement of decisions and commitments results in decisions 
being made at lover levels than perhaps they otherwise might be. One 
result of this is that program modifications can be made at these lower 
more isolated levels where their impact is less. More importantly, however, 
such a program composition is claimed to result in a higher level of abstrac- 
tion program. When a program has been built-up to an intermediate state of 
refinement, what has been written down is a suitable "common ancestor" 
for all possible programs which can be produced by further refinements. 

In other words, the structtire of the program is such as to anticipate its 
adaptations and modifications. As Dijkstra puts it, "The similarity 
between program modification and program composition is the similarity 
between the decision to be changed and the decision still left open". 

The starting point of the program composition is a concise statement of 
all of the things which the program is expected to do (e,g,, the highest 
level program specifications). Thereafter, one proceeds by conceiving 
a "computation" of "more primitive actions" that accomplish the desired 
net effect. If these more primitive actions belong to what Dijkstra 
calls "The weiJL understood repertoire" (e.g., they are computer executable 
instructions) then one is done. Otherwise, the process continues. At 
least four things about this approach are desirable from a program manage- 
ment point of view. 
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(l) The starting point is the program specification which provides 
mexiimam management visihility of the program development process 
from inception. 

The highest level requirements which are of prime interest to 
the program management and customer are addressed at program 
inception whereas decisions on detailed specifications (which 
are most subject to change) are delayed to the later stages 
of the development effort. This, of course, is the reverse 
of the traditional ’'hottom-up” development effort, 

{2) i&eh of the early stages of program design can. he performed 
via English language statements. Thus, early versions of 
the program will consist largely of computer instructions 
mechanizing the highest levels of control structure and a 
large number of English language statements (comments} which 
document in detail the decisions which have already been made 
Euid equally important, those still left open. Such a program 
is self -documenting to a very high degree. Also, there should 
be a very close correspondence uetween the program comments and 
the program specification document, 

(3) A running program is in existence virtually from program incep- 
tion, Integration is accomplished by adding refinements to 
the existing program in the top-down manner. Thus, integration 
is a continuous process performed throughout the development 
cycle. This contrasts strongly with the usual bottom-up 
development process where integration is the last and often 
a very traumatic step prior to final testing. 

(it) Program testing is also a continuous process performed throughout 
the development cycle, When refinements are added at any level, 
the program testing necessary to verify the requirements 
associated with these refinements is performed. There are three 
potential benefits of this approach. First, the higher level 
functional requirements are tested early in the development. 

This shoilLd tend to provide early reassurance to the customer 
that his most important requirements are being met. Second, 
testing at any level provides an important additional test 
of higher level code. Thus, the very important highest levels 
of code are exhaustively tested since they are exercised to 
some degree by the testing performed at all lower levels. 

Finally, the program itself acts as the "driver” for all testing. 
The need for separate "driver programs" to perform unit tests 
(as in the bottom-up approach) is eliminated. 

Before the reader gets the impression that "top down programming" and related 
design techniques are the answer to all the world's problems, one final 
point should be emphasized. This is simply that these design techniques 
are very difficult, especially the first time they are applied. The 
temptation to plunge into great detail in the "firm" areas rather than 
make hard decisions which are required in the "not so firm" areas must be 
resisted. Also, making the "right" decision is quite difficult. Subsequent 
developments may show clearly that a particular decision was wrong or 


that a particixlar decision siicnld have "been deferred and that a decision on 
a separate issue shouid have heen made instead* The capability to hacfeup 
and start again from a higher level must 'be present. The results of 
"errors” are out in the open for all to see and as a result no stigma 
should he attached to the individual who makes a decision which turns out 
to he vrong. In short, top down programming reqixires a tremendous change 
in management practice. 

’Whether or not this change can he made smoothly is yet to he seen. Probably 
the outstanding example of top down programming management is the IBM Chief 
Programmer Team experiment discussed in Section 2,2,2, The published 
accounts of this experiment claim results which can only be called outstandinh. 
Tempering all the enthusiasm, however, is the strong probability that the 
personnel involved in this experiment were of the very highest caliber. 

It remains to be seen if similar results can be attained by a team of more 
modest talents. What the Chief Programmer Team experiment may really be 
saving, is that outstanding personnel when highly motivated will produce 
outstanding results. 


D,2 CUERINT STATUS OF ST5UCTUHE33 PHOOEAMMIUG 

Structured programming was originally intended as a collection of programming 
disciplines which have in common the ohjectivii of producing reliable software. 
Over the years, this collection of ideas has fragmented euid today each of 
the major ideas previously discussed has become an entity in itself. 

The concept of a demonstration of program correctness has become an area 
of active research hut as yet little practical application. The concept 
of a limited number of control structures has been accepted by a substantial 
number of people and today it is often this concept that people refer to 
when they discuss "structured programming". Finally, the concept of top 
down programming has blossomed into a new software management philosophy 
of which the IBM Chief Programmer Team is the best known example. 

Because these ideas have developed along such separate paths, they are 
discussed in detail in three separate sections. Since program proving re- 
lies heavily on other aspects of "structured programming" it is presented 
last. 


D.2,1 Structured Coding 

The idea of coding a program using only a limited number of control structures 
is simplicity itself. Its theoretical basis is the theorem proved in the 
classic paper which states that any proper program segment can he 
flowcharted using only three control structures (Figure l) [Reference 3]. 

Of course, the fact that a program fan be so constructed is in no way a 
valid reason that a program should be so constructed. The arguments of 
Dijkstra and others have been generally accepted as valid arguments that 
coding should be restricted to a limited number of control structures, but 
the number and composition of such a set are subjects upon which there 
is virtually no agreement. 

The main problem stems from the fact that neither the IF-THM-ELSE nor the 
DO-TOIILE constructs are sufficiently general to satisfy a large number of 
programmers who are required to write "real world" programs. The IF-THEN- 
ELSE is not a general decision construct since it allows a choice between 
only two alternatives. The "case" construct (Figure 2) is a more general 
decision construct and the other construct shown in the figure is still 
more general. In a similar manner, the DO-VJHILB is not a general loop 
construct primarily because only a single exit from the loop is allowed, 

A more general loop construct is illustrated in Figure 3. 

A second practical objection to the three control structure limitation 
is the desired capability to exit from deeply nested code. There are 
two types of desired capability along these lines which are really quite 
different. The first is an immediate exit to a specified location in 
another subroutine. The first capability is easily mechanized with a 
GOTO statement whereas the second is a much more difficult capability to 
implement . 

Moat major languages implement a return to the Invoking subroutine (proce- 
dure) upon conclusion of the invoked subroutine (procedure) and no signi- 
ficant objection has arisen to including this construct within the realm 
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of "structured programming". Many languages , FOETRAU) also implement 

a return to the invoking subroutine from an interior point of the invoked 
subroutine. Objections to this construct are almost as common as the 
objections to the "GOTO", Since no major language includes a more general 
capability along these lines, the need for it is hard to Justify. Some 
limited capability along these general lines seems reasonable and is often 
cited as Justification for retaining use of the "GOTO" and the "Return". 

In theory, code can be structured in any programming language. The 
ease with which it can be performed, however, varies greatly from 
language to languti,^’- In assembly language, it is necessary to simulate 
the basic control structures. If a macro facility Is available, this 
can be done quite nicely by providing standard macros which mechanize 
the IF“THEN~ELSE and the DO-WHILE constructs. 

Several higher level languages have sufficient control structures to 
structure coding without modifying the language in any way, (PL/I, ALGOL 
and COBOL are examples). Even these languages, however, suffer from a 
lack of generality of the constructs available and/or a proliferation of 
optional methods of writing what is in essence the same construct. The 
FORTRAK language lacks an IF-TF'PiN-ELSE construct so to structure code 
in FORTRAN one must either construct an IF-THEN-ELSE type construct from 
more primitive operations (i.c-, the FORTRAN IF, CONTINUE and GOTO 
statements) or use some sort of FORTRAN language extension, A number of 
such FORTRAN language extensions have been proposed and several are 
operational. In these language extensions to "permit structured progreimming 
in FORTRAN", a proliferation of control struct' ' has occurred. Figure 4 
presents a representative cross section of con^^ux structtires which have 
been proposed as extensions to FORTRAN, 

Included is a flowchart of the construct, a typical "structxired coding" 
including indentation of the code for clarity and the construct as it 
might be mechanized in pure FORTRAN, Table 1 presents a summary of several 
existing systems which implement extensions to the FORTRAN language. 

Table 2 lists several characteristics ■which would be desirable features of such 
a system. 

The control structures given in Figure 4 fall into 4 categories: decision 

structures, single exit loops, two exit loops and miilti-exit loops. 

The inclusion of two exit and multi-exit loops probably requires some Justi- 
fication, A two exit loop is a desirable solution to a very common program- 
ming problem; i.e,, the processing loop in which one set of computations is 
to be performed if a certain operation is successful and an entirely 
different set of computations is to be performed if the operation is unsuccess- 
ful, Exffinples of this would be an iteration which either converged or 
did not and a search which either turned up a match or did not. Situations 
of this type can be handled by single exit loops; however, the coding is 
often awkward. Similar situations exist for which multi-exit loops are 
useful. These situations are not nearly so common. The primary Justifi- 
cation for the multi-exit loop is probably that it presents no more diffi- 
culty than the two exit loop. If one accepts two exit loops as a valid 
control structure for producing structured code, it is difficult to conceive 
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any reason for excluding the multi-exit loop. Discussions of multi-exit 

loop control structures are presented in References 6 and 7« T] 

li 

The number of control structures in each category and the diverse vays in 
vhich each control structure may he coded give some indication of the -- 

prohlems involved in standEirdizing the control structinres for structured ;; I 

coding. Some of the problems are (l) the concept of a block structure, 

( 2 ) hov to express the general decision control structure [Figure i ■>) how to ^ 

express the simple loop structure, and (4) what is the role of index i | 

variables • i 1 


The imposition of a block structiire is undoubtedly motivated by the modern 
trend to block stmctured languages. The reasons for this are rather 
subtle and have to do with the efficient mechanisation of certain advanced 
feat^ires (e,g,, recursive procedures and advanced data structures such as 
variable length strings, lists and stacks). The block structure is generally 
considered to be more desirable than the simpler subroutine structure of 
FOETRM for the mechanisation of such features (The FORTRAN language 
does not include such features so the issue really never arises). It 
remains to be seen whether there is a net advantage in imposing some form of 
block structiore on the FOR^ERAN language. There are, however, several 
systems which include this capability in some form. 

The fourth control structiire in Figure 4 is a very general decision control 
structure that appears in many structured coding systems. The literature, 
however, includes at least three different ways of viewing this construct 
(see the figure) which are suprisingly different. The first viewpoint 
iitvDoses a block structure such that the code for each alternative is contained 
within a separate block. The second viewpoint considers the construct to 
be an extension of the IF-THEN-ELSE statement. The add5,tion of an OEIF or 
similar statement converts the IF-THEN-ELSE into a general decision structure. 
The third viewpoint considers the construct as a generalisation of the simple 
CASE statement. The details of this generalisation are discussed in the 
note appended to construct #3 of Figure 4. 





The single exit loop structiire suffers from a proliferation of optional 
methods of writing code. There are two main problems: (l) where to place 

the test for exit from the loop and (2) the role of index variables. 

The DO-WHILE places the exit test at the beginning of the body of the loop 
code. This is a more general construct than the DO-UNTIL (which places the 
test at the end), since the body of the loop can he bypassed (i.e., executed 
aero times). The DO-UKTIL, however, often produces code which is easier 
to understand. For example, the code DO LOOP UNTIL Is=10 impacts the message 
that the loop is to he executed ten times somewhat better than the code 
DO LOOP WHILE I<11, Because these loop constructs fix the point at vhich 
the exit test is made, the loop control and termination test can be specified 
through an index variable (e.g., the variable I in the example code above). 
This saves the programmer the trouble of writing the code for the loop 
control parameters and the termination test. More importantly, however, 
this removes a possible soxurce of coding error. These advantages must he 
balanced against the disadvantage of lack of generality and flexibility. 
Construct shows a more general loop in the sense that the exit test can 
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be anywhere in the body of the loop. The price paid for this generality 
is that the programmer must define his OT-m loop control parameters and 
include code for the termination test. This in turn introduces potential 
sources of error. 

Similar constructs are possible for multi-exit loops. For exaiple, con~ 
structs #8 and #9 are the two exit equivalents of the DO-MILE and the 
DO-UNTIL* Multi-exit loops, however, are sufficiently complicated that 
the additional complexity introduced by requiring the programmer to code 
loop control parameters and termination tests seems minor. The general 
form of #10 would therefore seem more appropriate. 
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TABLE 1 

S0!'1E TOOLS FOE "STRUCTURED PROGRAf-ff^ING IN FORTRM" 


Neune 

Organization 

Control Structures 
{Figure U) 

Index 

Variable Options 

Other 

Features 

Limitations 

Status 

IFTRAJI 

General Research 

l»,5* 

1,6«» 

Programmer ’ s 
Responsibility 



Oper 

PREFOR 

IBM 

1,5,6,7,8,9 

Programmer * s 
Responsibility 

3 

S+a,l4d,Ue 


Oper 

FORTRAN 

Stanford Linear 

Accelerator 

Facility 

1.5,6 

Mans’' forms of 
DO loop 

8 


1 

Oper 

'**2RAMSF0R 

Boeing Computer 
Services 

1,2,3, U, 5,6 


Ua,ild 


Oper 

SC IF 

• 

l,2,ii,5,6 





STAPLE 

National Bureau 
of Standards 

Il.T* 

1,5,6»« 


4a,b,c,d 



PSST 

McDonnell Douglas 

^4,7, 10* 
1,5,6,8,9,10** 

Programmer ' s 
R espons ibility 

1,2,5,6,7 


Dev 


■•^Directly Available 

**Equivalent capability available 
as a special case of a more 
general construct. 


©*o 

l3^ ^ 


Other Featttres 

1, Automatic listing indentation. 

2, Preprocessor 

3 , Comniler 
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mi 




fg 

. Block Structure 
a. BKIN BLOCKS 

b. 

SELECT BLOCKS 


c . 

REPEAT BLOCKS 

s ^ 

d. 

EXIT BLOCK 


e. 

EXECUTE BLOCK 




5. Free Form Input 

6. Accents Pure FORTRAN 

7. Comments on executable 

8. Source Macros 


statements 
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TABLE 2 

DESIRABLE CHARACTERISTICS OP STRUCTURED PROGRAJ^ING AIDS FOR FORTRAN 


I 

^ 1. Easy to learn. A ralnimum number of additional control structures. There seems 

to "be general agreement that at least two are required (e.g., an IF-THEN-ELSE 
type construct and a loop structure which does not require statement numhers). 

TV 2. Mechanized indentation of listings. 

1 :^ 

^ 3. Free Form code input. In particular, should accept code in the indented form 

ji; (such as presented on the listing) hut not require it. 

j[: U. No change to the FORTRAN language - only additions. In particular, a pure FORTRAN 

K V 

program should pass through the preprocessor unchanged and execute properly. 

i 

" 5. Capability to include a comment on each statement for self-documentation purposes. 


1 


6. Full user control of Listing Format, delimiters, etc. 
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D,2,2 Program Management .nJecbniques 
D.2,2,1 Top Down Design Methodology 

The haaic idea of top down design was presented in Section 1.0, It was 
pointed out that top down design showed great promise hut also entailed 
some risk pince significant changes in management practice are required. 

This section discusses several areas of research which are attempting to 
develop guidelines and tools which may reduce the risk of implementing 
such a management strategy. 

The essence of top down programming is the division of a large program into 
a number of small ax~ subprograms (these subprograms may be subroutines, 
procedures, blocks, sjacros, etc,, depending on the programming language being 
used), Di^kstra suggests that there are at least four ways of conceiving 
subprograms , 

(1) Standard routines to be used as needed, 

(2) Objects to be conceived by the tiser to reflect his analysis, 

(3) A device for the reduction of program length, 

(4) A means for rebuilding a given machine (computer) into a mo* 
suitable one. 

Harlan Mills discusses the distinction between (l) and (2) above [Reference 8], 
’'First, we make a distinction between subprograms which are created for 
structuring the system and subprograms which carry out common low level 
processing functions in many places of the .system. The latter set of sub- 
programs we isolate first, and append to the programming language itself. 

Just as sine or exponential routines are regarded as part of PL/I or 
FORTRAN, These subprograms are documented and considered as part of the 
language description in which the programmers write the programming system”. 

Mills also discusses the importance of the subprogram as a means of reducing 
program length, ’’Imagine a one hundred page PL/I program written in "GO 
to" free code. Although it is highly structured, such a program is still 
not very readable. The extent of a major DO loop may be 50 to 60 pages, 
or an IF-THEN-ELSE statement may take up 10 or 15 pages. There is simply 
more than the eye can comfortably take in or the mind retain/* Mill's 
solution is to impose a discipline on the top down process such that each 
program segment is no larger than can he contained on a single page of 
computer printout, 

Dijkstra suggests that it is useful to view the subroutine as a means of 
rebuilding a given machine (computer) into a more suitable one. Starting 
at the top with the main program, Dijkstra chooses to view it as an entity in 
itself independent of the lower level subprograms. He views the main 
program as being executed by its own dedicated machine equipped with an 
adequate instruction repertoire (i.e,, each of the subprogram calls are 
available on this hypothetical machine as primitive instructions). In 
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actual practice, of course, this machine will not exist (Dijkstra calls this 
a virtual machine), The remainder of the programming task Dijkstra sees 
as programming the simulation of this "virtual" machine. The process 
is continued in a top down manner resxilting in a program arranged in 
"layers" or "levels". Each program level is to be understood all by 
itself, under the assumption of a suitable machine to execute it, while 
the function of each level is to simulate the machine that is assumed 
to be available on the level immediately above it.^ "The fact that a 
level contains "a bunch of programs" to be executed by some conceptual 
machine stresses the fact that the programs of this "bunch" are invited to 
share the same primitives." 

Dijkstra further elaborated on this process in Reference 9 when he presented 
a concrete example of his concept of "design by levels of abstraction". 

The example was a multiprogramming operating system for a university 
computer center (the "THE Operating System".) Dijkstra's team conceived 
the system design as existing in several levels each of which could be 
understood by itself as an entity. The lower level mechanized some very 
detailed, difficult and machine dependent tasks (e.g,, the real time 
clock and the interrupt structure). Above this level, however, these 
difficult concepts had in essence disappeared,. The very primitive operations 
involved had been replaced by primitives on a "higher level of abstraction". 
In this manner, the designers of the higher levels were freed from concern 
with lower level details which were irrelevant to the higher level designs, 

Barbara Liskow presented a design methodology which is based on structured 
programming in general and on "levels of abstraction" in particular 
[Reference lO], An abstraction is considered as expressing "what is 
being done without specifying how it is done," A level is defined not 
only by the abstraction which it supports but also by the resources it 
uses to realise the abstraction. Each level has resources (e.g, , I/O 
devices, data) which it owns exclusively and which other levels are not 
permitted to access. Each level is composed of a group of related functions 
of which there are two kinds; internal and external. Internal functions are 
used only within the level and cannot be referenced from other levels 
of abstraction. External functions may be referenced (called) only by 
higher level f\mctions. Lower levels are not aware of the existence of 
hi^er levels and therefore may not refer to them in any way. 

Thus, the programmer is encouraged to define subprograms for a variety 
of purposes (1) to "structure" his program, (2) to enhance clarity by 
reducing the length of other subprograms, (3) to rebuild his programming 
language into a language more suitable to his immediate needs and (U) to 
mechanize the internal and external functions of "levels of abstraction". 
This process is more art than science and as Mills puts it, "the 
programmer must use a sense of proportion and importance in identifying 
what is forest and what are the trees." Liskow presents a number of 
guidelines which may be useful in placing this rather nebulous process 
on a firmer basis [Reference 10], 
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C. A.R* Hoare presents a theory of data stmcturing on the premise that 
"It is necessary to introduce some convenient notation for expressing the 
abstractions involved" [Reference l]. An algorithmic langU-age is proposed 
whose purpose is to assist in the design, development and documentation 
of a program. This language is distinct from the programming language 
because "Some of the operations, although very helpful in the design of 
abstract programs may be very inefficient when mechanized directly on 
today's computers. An essential part of the program design process is 

to eliminate such operations in the transition from an abstract to a 
concrete program. The challenge of designing computers which can efficiently 
implement ever larger subsets of such a language may of course, be taken 
up in the future". 

The idea of a design language separate and distinct from a programming 
language is certainly not new. For example, the language of ordinary 
Eilgebra served this purpose for the FORTRAN programming language. The 
important point here is that an abstract problem solution should be developed 
using powerful data structures and operators pertinent to the problem 
being solved. The mind should not be constrained by the limitations {often 
severe) of the programming language being used. After the very difficult 
problems of what a program segment is to accomplish have been solved, most 
good programmers are quite adept at contriving an efficient method of 
implementation, 

D. 2,2,2 Chief Programmer Teams 

The IBM Chief Programmer Team experiment is documented in References 11 and 
12 and the material in this section relies heavily on these references. 

The experiment was the outgrowth of the work of Harlan Mills who has 
studied the conventional large, undifferentiated and relatively inexperienced 
team approach to programming projects and suggested that it might be 
replaced by a smaller, functionally specialized, and highly skilled team 
[Reference 13 1, The proposed organization is compared with a surgical team 
in which the chief prograsaaer is analogous to the chief surgeon and is 
supported by a team of specialists (as in a surgical team) whose members 
assist the chief rather than write parts of the program independently. 

Permanent members of a Chief Programmer Team are the Chief Programmer, the 
Backup Programmer and a Programming Librarian. The Chief Programmer is a 
senior level programmer who is responsible for the detailed development 
of a programming system. The Backup Programmer is also a senior level 
progranmer and works CIOS'- ly with the Chief Programmer to design and produce 
the system's key elements. The Backup Programmer has prime responsibility 
for system testing and also assumes the responsibility of the Chief 
Programmer should he leave the project. The Programming Librarian is re- 
sponsible for maintenance and operation of the Program Production Library 
used to keep all system programs and data both internally in machine 
readable form and externally in well organized, highly readable form. 

The legal, financial, administrative and reporting requirements are 
coordinated by a Project Manager assigned to the team. A9.so, a System 
Analyst is assigned to the team to assist as needed. Thereafter, the 
team is augmented by additional programmers who produce the remainder of 
the code under the close supervision of the Chief and Backup Programmer, 
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The Chief Programmer Team is intended to solve tvo probleri which are felt 
to be responsible for the relatively low productivity of current programming 
projects. 
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(1) Production projects are usually staffed by relatively 
inexperienced programmers at the working level and by 
more experienced programmers at the higher management 
levels. This results in several problems. First, 
the inexperience at the working level results in less 
than optimal design code and test. Second, the experienced 
programmers who have the insight and knowledge to correct 
this situation are in higher management positions where 
the administrative workload prevents them from effectively 
or economically performing any of the detailed work of 
programming. 

The Chief Programmer Team attempts to reintroduce the 
highly skilled programmer into the detailed production 
process but free him from both the details of programming 
trivia and the administrative workload. 

(2) In addition to normal programming activities such as 
design code and teat, a programmer normally spends a great 
deal of time with what are essentially purely clerical 
duties. For example, he must maintain his decks and 
listings, punch his own corrections, setup his runs and 
write status reports. 

The Chief Programmer Team attempts to free the programmer 
from a-1 .1 these clerical duties through the facilities of 
the Program Production Library and the Programming 
Librarian, The Program Production Library includes both 
machine and office procedures for performing the clerical 
duties of a programming project. 

The Program Produetion Library (PPL) comprises four partsi 
the machine-readable' internal library is a group of sub- 
libraries, each of which contain current project programming 
data. These data may be source code, relocatable modules, 
linkage-editing statements, object modules, job control 
statements, or test information. The status of the 
internal library is reflected in the human-readable external 
library binders that contain current listings of all library 
members and archives consisting of recently superseded 
listings. The machine procedures consist of standard computer 
steps for such procedures as the following j 

* Updating libraries 

* Retrieving modules for compilations and storing results 

* Linkage editing of jobs ax.d test runs 

* Backing up and restoring libraries 

* Producing library status listings 
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Office procedures are clerical rules used lay litrarians to 
perform the following duties: 

* Accepting directions marked in the external library 

* Using machine procedures 

* Filing updated status listings in the external library 

* Filing and replacing pages in the archives 

A prognammer using the PPL works only with the external library. 
Using standard conventions » he enters directly into the external 
library binders the changes to be made or work to be done. He 
then gives these changes to the librarian. Later he receives the 
updated external library binders, which reflect the new status 
of the internal library. The external library is always current 
and is organized to facilitate use by programmers, A chronological 
history of recent runs contained in the archive binders is retained 
to assist in disaster x-ecovery. 

The programmers are thus freed from handling decks, filing listings, 
keypunching, and spending unnecessary time in the machine area. 

By combining standard machine procedures, standard office 
procedures, and project libraries, the trained librarians provide 
a versatile programming service that allows a team to make more 
effective use of its time. 

The PPL also assists in improving productivity and quality by 
providing visibility of the work, there oy allo\ri.ng team members 
to be aware of the status of modules that they are integrating. 

Such visibility also permits members to be certain of interface 
requirements. The internal working language of a team are the 
code and statements in the libraries, rather than a separate 
set of documents that lag behind actual status. Programmers 
read each other's code in order to communicate definitions, 
interfaces, and details of operation. Only when a question 
arises that cannot be resolved by reading code is it necessary 
to consult another programmer directly, 

IBM selected the New York Times Information Bank as a project suitable for 
testing the validity of the Chief Programmer Team principles. This is an 
on-line system intended to replace the clipping file (morgue) used by the 
Times to provide background information for articles being written. 

The user views article abstracts selected by index terms and documents 
parameters (e.g,, date, section of the paper) vintil he has identified those 
articles most relevant to his immediate needs. The user may then view the 
entire text of the articles selected or request that a hard copy be made. 

The heart of the system is the conversational subsystem and its associated 
data base consisting of indexing data, abstracts and complete articles. The 
full text of all articles is stored on microfiche and made available to the 
system through four TV cameras contained in a microfiche retrieval device 
called the RISAR that was developed by FOTO-MEM, The rest of the data 
base is stored on disk. Other major system components are the Data Kntry Edit 
Subsystem, the File Maintenance Subsystem and five supporting subsystems. 
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Table 3 sxainmELrizes the soft-ware development tasks performed by the various 
members of the Chief Programmer Team assigned to the project. The numbers 
indicate the approximate sequence in which these tasks were performed by the 
various personnel. The order in which the tasks were performed was influenced 
by the desire to achieve a "running system" at an early date and also to achieve 
sufficient capability to begin building files at an early date. Otherwise, a 
top down approach -was generally employed. As can be seen, the integration and 
test functions were integral parts of the development process. 

Table h shows the staffing levels during the project. It is interesting 
to note the large amount of time charged by the senior personnel relative 
to the more junior teani members (one of the goals of the Chief Programmer 
Team approach) , This is certainly not typical of software development 
projects in general but some question remins as to whether this is 
characteristic of the Chief Programmer Team approach or whether it is due to 
some special characteristics of the particular project. Also of interest is 
the relatively short time spent on the project by most of the support personnel. 
This would seem* to indicate that a highly flexible staffing policy may be 
necessary to make the Chief Programmer Team function well. Once again, 
management wo-uld seem to he the determining factor in the success or failure of 
the Chief Programmer Team (to a much greater extent than in the traditional 
software development process), 

IBM has supplied several measures of programmer productivity and system 
quality achieved* Table 5 summarizes the results of acceptance testing 
and early operational experience -with regard to errors encountered. 

Table 6 summarizes the programmer productivity achieved in terms of source 
lines of code produced per programmer day. 
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TABLE U 

• IBM CHIEF PROGRA>ff-!ER TE.AM EXPERIf-JEriT 
Staff Time (*Ian Months) 


Work Tyne 



Chief 

Backup 

Analyst 

1 2 3 li 5 Technician Manager Sec’y 

Total 

Requirements 

2.5 

1.0 

8.0 

0.5 - - - “ - 

12.0 

Analysis 






System 

l+.O 

U.O 

14.5 

1,0. - - - - - - . _ 

13.5 

Design 






Unit design. 

12.0 

lU.O 

10.0 

13.0 4.5 2.8 3.T 4.5 - 

64.5 


programming , 
debugging and 
testing 


Documentation 

2.0 

2.0 

4.5 

1.5 

0.2 

0.2 

0.3 

0.3 

“ 

- 

- 

11.0 

Secretarial 

- 

- 

- 

- 

- 

■E5 


- 

- 

7.0 

T.O 

Librarian 

- 

- 

- 

- 

- 

- 

- 

5.5 

- 

2.0 

7.5 

Manager 

3.5 

2,0 

- 

- 

- 

~ 

- 

- 

11.0 


i6.5 

TOTAL 

24.0 

23,0 

2T.0 

l6.0 

4.T 

3.0 

4.0 

4.8 

5.5 

11.0 

9.0 
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TABLE 5 


IBM CHIEF PROGRAMMER TEAM EXPERIMEWT 
ERRORS IDENTIFIED DURING ACCEPTANCE TESTING 


Error 



File Maintenance 12,029 

0 

0 

0 

0 


Conversational 38,990 

9 

8 

3 

20 


Data Entry Edit 13,ijRl 

0 

0 

1 

1 


Other l8,881+ 

0 

0 

0 

0 


TOTAL 83,32*^ 

9 

8 

h 

21 



EliRORS IDENTIFIED DURING OPERATION 



* "INCORRECT FUNCTION" - refers to code which operated improperly. 

"OMITTED FUNCTION" - refers to specifications not implemented. 

"MISINTERPRETED - refers to code which did not perform precisely the 

FUNCTION" functions specified. 


TABLE 6 

IBI-1 CHIEF PR0GRAI®.1ER TEAM 15XPERIMENT - PROGRATMER PRODUCTIVITY 


Organigatlon 


Source lines per 
programmer day 


Unit design, programming, 65 

debugging, and testing 


All professional UT 

With librarian support I43 


Entire team 


35 


*The first row includes work done on unit design, coding, debugging, and 
acceptance testing. The second row summarizes professional work, which 
includes system design and documentation, but not librarian support. The 
third row includes all programming and librarian support. The last row 
presents the productivity of the entire team on the completed system 
(excluding requirements analysis). 


D,2,2,2 Ccanputer Program Management Technigue (CEMT) 

CPMT is a systematic discipline for managing the deYelopment, verification 
and documentation of scientific and engineering computer programs. It 
incorporates many of the design, coding eind testing concepts which have 
been developed through structured programming research. It also incorporates 
much of the Chief Programmer Team management philosophy although specific 
concepts have "been modified somewhat to mate them more compatible with the 
aerospace scientific-engineering environment, 

CPMT defines personnel assignments in terms of functions. Thus, for a 
small program, several functions may he performed hy the same person 
whereas for large programs a single function may require severeil people. 

The following functions are identified: 

Requestor - defines the program requirement in order to solve some 
specific prohlem. 

Study Manager - supervises the program development and ensures 
the implementation of CPM? procedures. 

Principal Investigator - responsible for the program meeting the 
technical requirements. 

Chief Programmer - responsible for the structural design of the 
program and the coding of high level or control components. 

Engineer/Programmer - responsible for coding, documentation and 
testing. 

Librarian - maintains the program workbook, coordinates all 
documentation and compiles project statistics (i.e,, actual 
versus projected results), 

CPt'OT identifies five phases of program development: 

(1) Planning - A study plan is prepared defining the proposed method 
of solution, potential problems, and schedule and cost estimates, 

(2) Design - The structural design of the program is developed, 

!IJest plans and acceptance criteria are also defined during this 
phase, 

(3) Development - The coding, subprogram documentation and unit 
testing of the program is performed in this phase, 

(U) Program Test and Verification - The complete program is tested 
during the phase and final documentation is prepared, 

(5) Release - A Program Manual and a CPMT Study Report are released. 

A document specifying baseline test cases is prepared, A 
configuration control manager for the program is assigned and 
the program is released to the customer. 
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CH>EP documentation is designed for ease of preparation. Skeleton documents 
are prepared as early as practical and details are added when they hecome 
available. Preparation of final reports is intended to be mainly an exercise 
of putting the finishing touches on working documentation which already 
exists. The clerical aspects of docvimentatlon are handled by the librarian 
so the programmer is freed to a large extent from document editing and 
rewrites. 

Management reviews are held throughout the program development process with 
formal reviews scheduled at the conclusion of each of the five phases 
discussed above. Figure 5 indicates the relationships between personnel 
functions, program phases, formal reviews and the more important documenta- 
tion requirements, CPMT procedures are described in detail in a proprietary 
document of McDonnell Douglas Corporation [Reference lit]. 
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Note: The CASE statement involves the choice of a particular path based on the value of a logical expression (P), 

the simplest form of which is: 

X R Y 

where X and Y are variables ( or expressions ) and R is a relation. 

In the simplest form of CASE statement iff 2) the first two of these quantities are assumed to be the same for all 
paths (e.g,, X is the variable I shown in the Figure, R is the relation "=” and a value of the variable Y is associated 
with each path). A more general form of the CASE statement (^3) allows both R and Y (but not X) to vary from path to 
path, (E.g., this form allows taking the first path if X is less than 6, the second path if X is greater than 10 and 
the third path if X is equal to 7.) The most general form of the CASE statement allows all three quantities 

(X, R and Y) to vary from path to path. 
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Appendix K 

PROVING PROGRAMS CORRECT 


E.l INTRODUCTION 

The first serious notion that programs could and should he proved correct is 
probably the pioneering work of Peter Naur (Reference l) and Robert Floyd 
(Reference 2). The methods propounded were quite similar and were developed 
independently. Naur's method was based on what he called "general snapshots" 
and was a rather informal (though rigorous) conception of a proof. Floyd's 
approach was somewhat more formal and several concepts fundamental to the 
modern formal proof methods are present in his paper. In particular l) the 
use of formal mathematical logic, 2) the idea of an "abstract program" and 
3) the idea of an "interpretation" of an abstract program. 

Program proving over the years has grown in two different directions which can 
probably be best described as fomial and informal. Informal program proving 
is most often encountered in the literature generally associated with structured 
programming whereas formal program proving is usually encountered in the litera- 
ture on artificial intelligence. The informal methods have the disadvantage 
that there are few underlying general principles and each problem presents a 
separate challenge. The advantage of the informal method is that the human 
"prover" may use any notation which fits the current problem and use any method 
of proof suitable to the particular problem at hand. The formal methods on the 
other hand have developed to the point where there is a quite solid mathe- 
matical basis. There are, however, two related and rather serious problems 
with the formal methods at the present time. 

The formal methods require that either a simple language (e.g., the first order 
predicate calculus) or a complex language (e.g., second or higher order mathe- 
matical logic) be used for the mechanics of the proof. If the simple language 
is chosen there are reasonably efficient proof methods available but one en- 
counters extreme difficulty in formixlating "real" problems because certain basic 
mathematical concepts (most notably the equality relation) axe not easily ex- 
pressed in this language. If the complex language is chosen, the formulation 
problem is eased considerably but there are as yet no really satisfactory proof 
methods available. Before a really practical application of the formal proof 
methods can be made it is probable that greatly improved proof methods for 
higher order logics will be required, 

E.2 INFORMAL PROOF METHODS 

The literature on informal proofs of programs consists almost entirely of samule 
demonstrations for particular algorithms. At least two different approaches 
are identifiable; the constructive approach and the verification approach. 

In the constructive approach (Dijkstra, Reference 3) an algorithm proof is 
developed in a top down manner from the algorithm specifications. The steps of 
the algorithm proof are then converted to executable code in what is normaVly 
a relatively trivial exercise. The result is an algorithm which has been 'pr-ivcd 
correct" and then converted to executable code. It is important to note that it 
is the algorithm not the code that has been proved correct. 
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In the verification approach (Floyd, Reference 2) the executable code is 
assumed to exist. The algorithm proof steps referred to above either exist or 
must be generated. Figure E-1 illustrates the verification problem given the 
code in flow chart form and the algorithm proof steps in the form of state- 
ments in mathematical logic (propositions). 
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Figure E-1. 


Flowchart Illustrating Floyd's Method of Program Verification 
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The code "betweea propositions Floyd c^Us "commands." On the flowcharts these 
commands are connected hy "ainrows" representiiag the possible passages of control 
between the commands . Bach command (except START and HART) has at least one 
"entrance" arrow (a^ ) and at least one "exit" arrow (b^ ) . A "proposition" is 
associated with each of these arrows. Thus each command has one or more entrance 
propositions (Pj) and one or more ecit propositions (Qj). Using this terminology 
Floyd defines a v^ification as "a proof that for every command c of the flow- 
chart, if control should enter the command by an entrance (a^) with Pi true, then 
control must leave the command, if at all, by an «cit (bO with Qj time." The 
entrance and exit propositions Floyd calls the "verification conditions" for a 
command. These verification conditions are identical to the "algorithm proof 
steps" generated in the constructive approach. Thus "verification" bridges the 
gap between an algorithm proof and a proof of its representation in executable 
code. 

As the reader has probably determined, proving a program correct requires two 
very difficult steps: l) determining "verification conditions" which faithfully 

represent the desired algorithm, and 2) performing the proof required by the 
above definition of a verification. Unfortunately the literature offers little 
guidance except by specific example. One exception to this is Hoare’s concept 
of Invariants (References U and 5) which appears to offer solid foundations in an 
area where few exist. 

Invariants 


Hoare (Reforence iv) defines an invariant as "a formula of logic which is in- 
tended to remain true throughout the execution of a program segment" (even 
though the values of any variable appearing in the formula may be changed by 
the execution). One reason that invariants are important is because they provide 
a very useful insight into how a loop performs a desired function. To be used 
in this manner, an invariant is required whose meaning is essentially a specifi- 
cation of what the loop is intended to accomplish. Formulation of the specifica- 
tions of a program segment (e.g., a loop) in invariant form is a step which 
sometimes requires great ingenuity. The basic idea, however, can be illustrated 
by the simple exEunple shown in Figure E-2, The key step is the expression of 
the program specifications in invariant form. In this simple example this is 
accomplished by replacing the parameter N (which is important only to a final 
result) by the parameter J (which has significance for all intermediate results). 
It should be noted that upon termination J = N and the two specifications are 
the same. The invariant form however, is true throughout execution of the loop 
especially at the points labeled © © (3) and (© in Figure E-2, The non- 
invariant form is necessarily true only at point (S) . 
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ProlJlem : Find the maximum value of an array A of dimension N. 

equal to this value. 

RroKram Segment SneclflGation: 

For all I such that (l-I-N) B-A[l] 

For at least one I such that (l5l5lf) B « A[l] 

Program Seppient Specification In Invariant Form : 

For all I such that (l-I-j) B-AEI] 

For at least one I such that (l-I-J) B - A[I] 

For loop Termination J = N 

Pi’ogram: 



Figure E-2, Program Specification With Invariants 





Once the invariant for a loop has "been determined, the formal proof of loop 
correctnesB is straightforward. The idea is to prove that the invariant is 
true upon exit from the loop (i.e., at point (§) in Figure E-2), This is 
done "by two steps which are; 

(1) Prove the invariant is true before the loop is entered 
(i.e., at point @ in Figure E-2). 

(2) (a) assume the exit condition is not satisfied (i.e., KH) 

(h) assume the invariant is true at point 

(c) mentally execute the body of the loop once, i.e., 

(J<-J+1; IF (B<A[J]) THEN B -^A[J] ) 

(d) prove that the invariant remains true. 

(i.e., that it is true at @ ) 

The above two steps and the principle of mathematical induction are sufficient 
to prove the desired result - namely that the invariant is true upon exit 
from the loop. Loop termination is proved separately and will establish that 
upon exit J=II which makes the Invariant form of program specification identical 
to the original program specification. 

E.3 FORMAL PROGRAM PROVING 

Formal program proving attempts to overcome a very serious drawback of the in- 
formal methods - i.e., the necessity of dealing with each program on an indivi- 
dual basis. To do this it is necessary to abstract the concept of a program - 
to identify the essential "structure” of a program and to eliminate the details 
which are peculiar to a certain representation of that program. The result is 
a "program schema" or "abstract program" which is a sort of sfceleton program 
consisting solely of assigument statements and branch statements. More speci- 
fically an abstract program consists of the following; 

(1) A vector of input variables x 

(2) A vector of program variables ^ 

(3) A vector of output variables z 

(4) A vector of program constants a. 

(5) Assignment statements of the form 

ifH ( x , n 1 

(6) Branch statements consisting of a predicate (logical expression) 
Pj(x,t/) where either of two paths are taken depending on the 
truth of falsity of P^(x,^). 
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Actual input and output do not occur in an aTastract program. Rather this is 
handled hy assignment statements (i.e., input is accomplished by assigning 
a function of input variables to a vector of program variables, e.g., [x]. 

Similarly output is accomplished by assigning a function of input and program 
variables to a vector of output variables . Figure E-3 shows an 

abstract program in flowchart form. 

An "interpretation" of an abstract program specifies 

(1) Specific functions and predicates 

(2) Specific values for all program constants 

( 3 ) The domains of the input, program and output variables, (Note 
in particular that values for input variables are not assigned - 
merely the domain - e.g., an input variable may be constrainted 
to be a positive integer but its value is unspecified). 

Under an "interpretation" an "abstract program" becomes an actual program capable 
of execution once the values of the input variables are specified. Thus an 
"interpretation" forms the link between abstract programs and actual executable 
programs . 

The theoretical basis for the formal approach is due to Manna (Reference 6) 
who showed in essence that the verification of any abstract program can be 
converted into the proving of a theorem (usually) in the first order predicate 
calculus. The development which follows is based on Reference 7 and to a limited 
extent assumes the reader is familiar with the predicate calculus (Reference 8, 
chapter 6 is a reasonably straightforward development of the predicate calculus 
for the reader desiring more background). Before proceeding with the formalism 
however, it is necessary to firm up some basic concepts. 

Roughly speaking, a program may be said to be correct if its execution terminates 
and it yields the desired final result. However, since both termination of 
execution and attainment of a desired result usually depend on the input vector 
(x), it is necessary to introduce a predicate i{)(x) representing these constraints. 
Likewise, it is necessary to formalize the idea of "attaining the desired final 
result" by introducing a predicate ijj(x,z] which is true if and only if z is the 
desired output for valid input X. Thus we may speak of a program being correct 
with respect to the input predicate and the output predicate i|;{x,z). 

It has been found useful to define two types of program correctness primarily 
because a proof of program termination is often (but not always) most easily 
performed separate from the proof of correctness. Thus a program is said to be 
correct with respect to input predicate tj>(x) and output predicate i{»(x,z) if it 
yields the correct answer (i.e., satisfies if>(x,z]j and if it terminates for all 
valid input (i.e., input vectors X satisfying ^{x] ] . Alternatively, a program 
is said to be partially correct with respect to input predicate (fi(x) and output 
predicate ^i;fx,z) if it yields the correct answer when it terminates (for valid 
input x). A proof of termination together with a proof of partial correctness 
is of course equivalent to a proof of correctness. 
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Manna's formalism assumes an alastract program to consist of a series of state- 
ments of the form 

I: IF 

THEN 

GOTO 

ELSE 

GOTO Ig 

where I; Ig are statement labels 

Pj^(jv,f/) is a preflleate 
1 2 

{x,(/j are functions 

In the above standard form, any of the go to statements may be replaced by the 
HAIiT command which indicates program termination. It is a relatively straight- 
forward exercise to convert any abstract program to this standard form. 

With each statement in the above standard form. Manna associates a ’’well formed 
formula" in the predicate calculus; ("well formed formula" is essentially a 
statement expressed in mathematical logic which is either true or false depending 
on the values of the variables contained in it). 

\ = Vy. d^U,y)^ 

IF Pj(x,£/) 

THEN (X,j(^^(x,y)) 
else q- Cx,j(.^(x.£/)) 

Ig X 

where V is read "for all y" 

y 

means "implies" 

th 

and P^(x,i/) is the predicate associated with the i statement of the abstract 
^program. 

^i’^^il’^a "verification conditions" aa defined by Floyd which 

are associated with the i statement of the abstract program. 
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A Talock diagram of the i “ statement of the abstract program is given in Pigpre 
E-4, showing the "Floyd verification conditions," If one of the go to state- 
ments is replaced hy HALT, the corresponding verification condition is replaced 
hy the output predicate ijj(x,z). 

The formalism continues by defining two additional well formed formulas 
T(x.) = g^(x,r/)Avr^A., . 

T(x) = T(x) with the output predicate i|>(x,a) replaced by its 
coraplonent (i.e., '^i|j[x,2)) wherever it appears. 

In the above, A means logical "and" 

'Vt means logical "not" 

Finally, the desired result is the two well formed formulas: 

= V {<}){x) =^t(x.)} 

P * 

These formulas form the basis for Manna’s two theorems (proved in Reference 6): 

Theorem 1: The program is partially correct with respect to 

and if and only if is satisfiable 

(i.e., is true under some "interpretation" of the 
Floyd verification conditions). 

Theorem 2: The program is correct with respect to <}> and tp 

if and only if Wp[(fi,ijj] is unsatisfiable (i.e., 
is false under every "interpretation" of the 
Floyd verification conditions). 

Proving satisfiability (or unsatisfiability) of a well formed formula in the 
predicate calculus is a very complex topic and is not discussed here. The 
interested reader is referred to References T and 8 for an introduction to 
the topic. 
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